Constructor Detail

Method Detail

• map

   final static <I extends Any, O extends Any> PCollection<O> map(PCollection<I> $self, String name, SerializableFunction<I, O> transform)

  Extension for a map operation on a PCollection.

  inputTeamsPCollection
      .map("Step name") { team -> TestSettings.toOtherTeam(team) }

  Parameters:

  name - pipeline step name

  transform - current transformation function

• flatMap

   final static <I extends Any, O extends Any> PCollection<O> flatMap(PCollection<I> $self, String name, SerializableFunction<I, Iterable<O>> transform)

  Extension for a flatMap operation on a PCollection.

  inputTeamsPCollection
      .flatMap("Step name") { team -> team.players }

  Parameters:

  name - pipeline step name

  transform - current transformation function

• mapFn

   final static <I extends Any, O extends Any> PCollection<O> mapFn(PCollection<I> $self, String name, SerializableFunction<I, O> transform, SerializableAction setupAction, SerializableAction startBundleAction, SerializableAction finishBundleAction, SerializableAction teardownAction)

  Extension for a map operation in a PCollection with DoFn lifecycle functions. Via this method, while the transformation is done, some methods can be passed to interact with DoFn lifecycle methods (setup, startBundle, finishBundle, teardown).

  inputTeamsPCollection
      .mapFn(
          "Step name",
          { team -> TestSettings.toOtherTeam(team) },
          setupAction = { print("Test setup action") },
          startBundleAction = { print("Test start bundle action") },
          finishBundleAction = { print("Test finish bundle action") },
          teardownAction = { print("Test teardown action") },
      )
      .result

  Parameters:

  name - step name

  transform - current transformation function

  setupAction - setup action function

  startBundleAction - start bundle action function

  finishBundleAction - finish bundle function

  teardownAction - teardown action function

• flatMapFn

   final static <I extends Any, O extends Any> PCollection<O> flatMapFn(PCollection<I> $self, String name, SerializableFunction<I, Iterable<O>> transform, SerializableAction setupAction, SerializableAction startBundleAction, SerializableAction finishBundleAction, SerializableAction teardownAction)

  Extension for a flatMap operation in a PCollection with DoFn lifecycle functions. Via this method, while the transformation is done, some methods can be passed to interact with DoFn lifecycle methods (setup, startBundle, finishBundle, teardown).

  inputTeamsPCollection
      .flatMapFn(
          "Step name",
          { team -> team.players },
          setupAction = { print("Test setup action") },
          startBundleAction = { print("Test start bundle action") },
          finishBundleAction = { print("Test finish bundle action") },
          teardownAction = { print("Test teardown action") },
      )
      .result

  Parameters:

  name - step name

  transform - current transformation function

  setupAction - setup action function

  startBundleAction - start bundle action function

  finishBundleAction - finish bundle action function

  teardownAction - teardown action function

• mapFnWithContext

   final static <I extends Any, O extends Any> PCollection<O> mapFnWithContext(PCollection<I> $self, String name, SerializableFunction<DoFn.ProcessContext<I, O>, O> transform, SerializableAction setupAction, SerializableAction startBundleAction, SerializableAction finishBundleAction, SerializableAction teardownAction, Iterable<PCollectionView<?>> sideInputs)

  Extension for map operation on PCollection with DoFn lifecycle functions. In this case, the current function for transformation has the DoFn.ProcessContext as input. So, via this feature we can access to process context of the current transformation. For example DoFn.ProcessContext give the possibility to retrieve side inputs while applying the transformation logic.

  Via this method, while the transformation is done, some methods can be passed to interact with DoFn lifecycle methods (setup, startBundle, finishBundle, teardown).

  Side inputs as PCollectionView can be passed to this DoFn.

  inputTeamsPCollection
       .mapFnWithContext(
           "Step name",
           { context: DoFn<Team, OtherTeam>.ProcessContext -> toOtherTeamWithSideInputField(sideInput, context) },
           setupAction = { print("Test setup action") },
           startBundleAction = { print("Test start bundle action") },
           finishBundleAction = { print("Test finish bundle action") },
           teardownAction = { print("Test teardown action") },
           sideInputs = listOf(sideInput)
       )
       .result
  
  fun toOtherTeamWithSideInputField(
      sideInput: PCollectionView<String>,
      context: DoFn<Team, OtherTeam>.ProcessContext
  ): OtherTeam {
  
      val inputTeam: Team = context.element()
      val otherTeam = TestSettings.toOtherTeam(inputTeam)
  
      otherTeam.sideInputField = context.sideInput(sideInput)
  
      return otherTeam
  }

  Parameters:

  name - pipeline step name

  transform - current transformation function

  setupAction - setup action function

  startBundleAction - start bundle action function

  finishBundleAction - finish bundle action function

  teardownAction - teardown action function

  sideInputs - side inputs associated to this DoFn class

• flatMapFnWithContext

   final static <I extends Any, O extends Any> PCollection<O> flatMapFnWithContext(PCollection<I> $self, String name, SerializableFunction<DoFn.ProcessContext<I, O>, Iterable<O>> transform, SerializableAction setupAction, SerializableAction startBundleAction, SerializableAction finishBundleAction, SerializableAction teardownAction, Iterable<PCollectionView<?>> sideInputs)

  Extension for flatMap operation on PCollection with DoFn lifecycle functions. In this case, the current function for transformation has the DoFn.ProcessContext as input. So, via this feature we can access to process context of the current transformation. For example DoFn.ProcessContext give the possibility to retrieve side inputs while applying the transformation logic.

  Via this method, while the transformation is done, some methods can be passed to interact with DoFn lifecycle methods (setup, startBundle, finishBundle, teardown).

  Side inputs as PCollectionView can be passed to this DoFn.

  inputTeamsPCollection
       .flatMapFnWithContext(
           "Step name",
           { context: DoFn<Team, Player>.ProcessContext -> toPlayers(sideInput, context) },
           setupAction = { print("Test setup action") },
           startBundleAction = { print("Test start bundle action") },
           finishBundleAction = { print("Test finish bundle action") },
           teardownAction = { print("Test teardown action") },
           sideInputs = listOf(sideInput)
       )
       .result
  
  fun toPlayers(
       sideInput: PCollectionView<String>,
       context: DoFn<Team, Player>.ProcessContext
  ): List<Player> {
     // Get side input field.
     val sideInputField: String = context.sideInput(sideInput)
  
     val inputTeam: Team = context.element()
     val players = inputTeam.players
  
     // Can add logic based on side input field....
  
     return players
  }

  Parameters:

  name - pipeline step name

  transform - current transformation function

  setupAction - setup action function

  startBundleAction - start bundle action function

  finishBundleAction - finish bundle action function

  teardownAction - teardown action function

  sideInputs - side inputs associated to this DoFn class

• filter

   final static <I extends Any> PCollection<I> filter(PCollection<I> $self, String name, SerializableFunction<I, Boolean> transform)

  Extension for filter operation on a PCollection.

  inputTeamsPCollection
       .filter("Step name") { team: Team -> this.isNotBarcelona(team) }
       .result

  Parameters:

  name - pipeline step name

  transform - current transformation function