Package io.activej.aggregation

Class Aggregation

java.lang.Object
io.activej.aggregation.Aggregation
All Implemented Interfaces:
IAggregation, io.activej.common.initializer.WithInitializer<Aggregation>, io.activej.eventloop.jmx.EventloopJmxBean, io.activej.eventloop.jmx.EventloopJmxBeanWithStats
public class Aggregation extends Object implements IAggregation, io.activej.common.initializer.WithInitializer<Aggregation>, io.activej.eventloop.jmx.EventloopJmxBeanWithStats
Represents an aggregation, which aggregates data using custom reducer and preaggregator. Provides methods for loading and querying data.

  • Field Details

    • DEFAULT_CHUNK_SIZE

      public static final int DEFAULT_CHUNK_SIZE
      See Also:

    • DEFAULT_REDUCER_BUFFER_SIZE

      public static final int DEFAULT_REDUCER_BUFFER_SIZE
      See Also:

    • DEFAULT_SORTER_ITEMS_IN_MEMORY

      public static final int DEFAULT_SORTER_ITEMS_IN_MEMORY
      See Also:

    • DEFAULT_MAX_INCREMENTAL_RELOAD_PERIOD

      public static final Duration DEFAULT_MAX_INCREMENTAL_RELOAD_PERIOD

    • DEFAULT_MAX_CHUNKS_TO_CONSOLIDATE

      public static final int DEFAULT_MAX_CHUNKS_TO_CONSOLIDATE
      See Also:

  • Method Details

    • create

      public static Aggregation create(io.activej.eventloop.Eventloop eventloop, Executor executor, DefiningClassLoader classLoader, AggregationChunkStorage aggregationChunkStorage, io.activej.csp.process.frames.FrameFormat frameFormat, @NotNull @NotNull AggregationStructure structure)
      Instantiates an aggregation with the specified structure, that runs in a given event loop, uses the specified class loader for creating dynamic classes, saves data and metadata to given storages. Maximum size of chunk is 1,000,000 bytes. No more than 1,000,000 records stay in memory while sorting. Maximum duration of consolidation attempt is 30 minutes. Consolidated chunks become available for removal in 10 minutes from consolidation.
      Parameters:
      eventloop - event loop, in which the aggregation is to run
      executor - executor, that is used for asynchronous work with files
      classLoader - class loader for defining dynamic classes
      aggregationChunkStorage - storage for data chunks
      frameFormat - frame format in which data is to be stored

    • withChunkSize

      public Aggregation withChunkSize(int chunkSize)

    • withReducerBufferSize

      public Aggregation withReducerBufferSize(int reducerBufferSize)

    • withSorterItemsInMemory

      public Aggregation withSorterItemsInMemory(int sorterItemsInMemory)

    • withMaxIncrementalReloadPeriod

      public Aggregation withMaxIncrementalReloadPeriod(Duration maxIncrementalReloadPeriod)

    • withIgnoreChunkReadingExceptions

      public Aggregation withIgnoreChunkReadingExceptions(boolean ignoreChunkReadingExceptions)

    • withMaxChunksToConsolidate

      public Aggregation withMaxChunksToConsolidate(int maxChunksToConsolidate)

    • withTemporarySortDir

      public Aggregation withTemporarySortDir(Path temporarySortDir)

    • withStats

      public Aggregation withStats(AggregationStats stats)

    • getStructure

      public AggregationStructure getStructure()

    • getState

      public AggregationState getState()

    • setState

      public void setState(AggregationState state)

    • detachState

      public AggregationState detachState()

    • getKeys

      public List<String> getKeys()

    • getMeasures

      public List<String> getMeasures()

    • getKeyTypes

      public Map<String,FieldType> getKeyTypes()

    • getMeasureTypes

      public Map<String,FieldType> getMeasureTypes()

    • getPartitioningKey

      public List<String> getPartitioningKey()

    • aggregationReducer

      public <K extends Comparable, I, O, A> StreamReducers.Reducer<K,I,O,A> aggregationReducer(Class<I> inputClass, Class<O> outputClass, List<String> keys, List<String> measures, DefiningClassLoader classLoader)

    • consume

      public <T, C, K extends Comparable> StreamConsumerWithResult<T,AggregationDiff> consume(Class<T> inputClass, Map<String,String> keyFields, Map<String,String> measureFields)
      Provides a StreamConsumer for streaming data to this aggregation.
      Type Parameters:
      T - data records type
      Parameters:
      inputClass - class of input records
      Returns:
      consumer for streaming data to aggregation

    • consume

      public <T> StreamConsumerWithResult<T,AggregationDiff> consume(Class<T> inputClass)

    • estimateCost

      public double estimateCost(AggregationQuery query)

    • query

      public <T> StreamSupplier<T> query(AggregationQuery query, Class<T> outputClass)

    • query

      public <T> StreamSupplier<T> query(AggregationQuery query, Class<T> outputClass, DefiningClassLoader queryClassLoader)
      Returns a StreamSupplier of the records retrieved from aggregation for the specified query.
      Specified by:
      query in interface IAggregation
      Type Parameters:
      T - type of output objects
      Parameters:
      query - query
      outputClass - class of output records
      Returns:
      supplier that streams query results

    • getNumberOfOverlappingChunks

      public int getNumberOfOverlappingChunks()

    • consolidateMinKey

      public io.activej.promise.Promise<AggregationDiff> consolidateMinKey()

    • consolidateMinKey

      public io.activej.promise.Promise<AggregationDiff> consolidateMinKey(Set<Object> lockedChunkIds)

    • consolidateHotSegment

      public io.activej.promise.Promise<AggregationDiff> consolidateHotSegment()

    • consolidateHotSegment

      public io.activej.promise.Promise<AggregationDiff> consolidateHotSegment(Set<Object> lockedChunkIds)

    • consolidate

      public io.activej.promise.Promise<AggregationDiff> consolidate(List<AggregationChunk> chunks)

    • getChunksForConsolidation

      public List<AggregationChunk> getChunksForConsolidation(Set<Object> lockedChunkIds, boolean hotSegment)

    • getChunkIds

      public static String getChunkIds(Iterable<AggregationChunk> chunks)

    • getMaxIncrementalReloadPeriod

      public Duration getMaxIncrementalReloadPeriod()

    • setMaxIncrementalReloadPeriod

      public void setMaxIncrementalReloadPeriod(Duration maxIncrementalReloadPeriod)

    • getChunkSize

      public int getChunkSize()

    • setChunkSize

      public void setChunkSize(int chunkSize)

    • getSorterItemsInMemory

      public int getSorterItemsInMemory()

    • setSorterItemsInMemory

      public void setSorterItemsInMemory(int sorterItemsInMemory)

    • isIgnoreChunkReadingExceptions

      public boolean isIgnoreChunkReadingExceptions()

    • setIgnoreChunkReadingExceptions

      public void setIgnoreChunkReadingExceptions(boolean ignoreChunkReadingExceptions)

    • getMaxChunksToConsolidate

      public int getMaxChunksToConsolidate()

    • setMaxChunksToConsolidate

      public void setMaxChunksToConsolidate(int maxChunksToConsolidate)

    • getConsolidationSeconds

      @Nullable public @Nullable Integer getConsolidationSeconds()

    • getConsolidationLastTimeSeconds

      @Nullable public @Nullable Integer getConsolidationLastTimeSeconds()

    • getConsolidations

      public int getConsolidations()

    • getConsolidationLastError

      public Exception getConsolidationLastError()

    • getChunks

      public int getChunks()

    • getStats

      public AggregationStats getStats()

    • getEventloop

      @NotNull public @NotNull io.activej.eventloop.Eventloop getEventloop()
      Specified by:
      getEventloop in interface io.activej.eventloop.jmx.EventloopJmxBean

    • toString

      public String toString()
      Overrides:
      toString in class Object