Skip to content

Releases: Journera/glutil

v1.0.0 - Support for arbitrary partitioning schemas

29 Oct 18:32
565ea32
Compare
Choose a tag to compare

Where the initial releases of glutil only supported tables with a [year, month, day, hour] schema, now arbitrary partition schemas are supported. So long as they conform to either:

  • Partition keys are path segments

    partition keys: [year, month, day]
    s3://bucket/table/prefix/2019/08/12/ => [2019, 08, 12]
    
  • Partition keys are path segments, with hive-format paths

    partition keys: [dt]
    s3://bucket/table/prefix/dt=2019-08-12/ => [2019-08-12]
    
  • Single-key partitions, where the partition value is the path with slashes replaced with hyphens

    partition keys: [dt]
    s3://bucket/table/prefix/2019/08/12/ => [2019-08-12]
    

Everything appears to be working as expected, the only change is that Partitioner.partitions_on_disk only accepts a limit_days key if the table's first three partition keys are [year, month, day].

v0.1.3

24 Jun 20:57
Compare
Choose a tag to compare
v0.1.3