dbt - stage_external_sources - partitioning

Question

I'm trying to do a nice partitioning of the incremental files delivered on azure blob storage.

However, I can't get partition column and table columns to play nice. If I remove partition column, I get all table columns specified. If I have partition column, I only get variant column and partition column.

  - name: arsm
    external:
      location: '@my_azure_stage'
      file_format: 'myformat'
      pattern: '.*path.*tablename_.*'
      auto_refresh: false # depends on your Azure setup
      partitions:       # optional
        - name: LOAD_DATE
          data_type: date
          expression: TO_DATE(substring(metadata$filename,16,10))
      columns:
      - name: "AoNr" 
        data_type: bigint
      - name: "AoNrAlfa" 
        data_type: varchar(65)
      - name: "AoPos" 
        data_type: int
      - name: "ArtikelVariant" 
        data_type: varchar(30)
      - name: "ArtKalkBer" 
        data_type: NUMERIC

Have you confirmed that myformat correctly parses the file? — Nat Taylor

Jeremy Cohen Jeremy Cohen · Accepted Answer · 2021-03-23T14:17:56

I think you just need to out-dent the columns array by two spaces. The columns array should be a top-level key of the source table, at the same level as name and external; right now you have it nested within the external dict.

dbt - stage_external_sources - partitioning

1 Answers