2025-11-13
Forum: Meeting - EG Flow - Data _ Azure Blob, Queue and Tables _ Design Discussions
Topics: Use Azure Blob Storage with hierarchical file structure instead of Table Storage for all data persistence.
Storage Structure
Containers:
| Container Name | Description |
|---|---|
| {org-id}-documents-{year} | Stores invoices and documents with multiple format versions (PDF, HTML, JSON) |
| {org-id}-batches-{year} | Stores batch processing data and metadata |
| {org-id}-data | Stores organization configuration and template files |
Blob Storage Structure
| Container | Blob Name | Description |
|---|---|---|
| {org-id}-documents-{year} | /invoices/{invoice-id}.pdf | Generated invoice in PDF format |
| {org-id}-documents-{year} | /invoices/{invoice-id}.html | Generated invoice in HTML format |
| {org-id}-documents-{year} | /invoices/{invoice-id}.json | Invoice metadata and data in JSON |
| {org-id}-documents-{year} | /documents/{document-id}.pdf | Document in PDF format |
| {org-id}-documents-{year} | /documents/{document-id}.html | Document in HTML format |
| {org-id}-documents-{year} | /documents/{document-id}.json | Document metadata and data in JSON |
| {org-id}-batches-{year} | /batches/{batch-id} | Batch processing information |
| {org-id}-data | organization.json | Organization configuration and settings |
| {org-id}-data | /templates/{template-name}.html | Document/invoice templates |
Rationale
Eliminates Table Storage partition key complexity and cross-partition query costs
Enables year-based partitioning for lifecycle management
2025-11-11
**Forum:** Biweekly Flow Meeting
**Topics:** Organization identifier, ID tools
**Organization identifier & ID tools**
We are not going to use the legal market identifier known as a "organizational number", we will use our own identifying unit - NOT auto incremental as with a relational database.
What was discussed was the probability of a `UUID` (Universally Unique Identifier) or `LUUID` (Lexical Universally Unique Identifier). The latter being easier to read and indexed.
**Other**
* Print Partners - There will be more than one partner available, but EG will choose which one to use for the organization.
* The `PUT`-method for the organization endpoint was thought to state that the organization has uploaded their data and that the service could trigger the batch process.
## 2025-10-27
**Forum:** Biweekly Flow Meeting
**Topics:** Logging tools, Introduce basic architecture from Axel, structure changes
Project solution will change from _"EGU.EGFlow"_ to _"EGU.Flow"_.
And additionally we will be having console applications to process batch, naming conventions will follow from the above structure.
In regards to logging EG Flow will be using azure application insights for logging. Seri logs for structuring the logs.
```
EGU.Flow/
├── EGU.Flow.AppHost/ # Aspire AppHost project
│ └── Program.cs # Configures serviceDefaults and service wiring
│
├── EGU.Flow.Core/ # Shared contracts and types
│ ├── DTOs/
│ ├── Enums/
│ ├── Interfaces/
│ └── Common/
│
├── EGU.Flow.Domain/ # Domain models and logic (used by API)
│ ├── Models/
│ └── DomainServices/
│
├── EGU.Flow.BusinessLogic / # Business logic
│ ├── Services/
│ └── Mappers/
│
├── EGU.Flow.CoreApiService/ # ASP.NET Core Web API
│ ├── Controllers/
│ └── Program.cs
│
├── EGU.Flow.Web/ # ASP.NET Core Web App (e.g.,Blazor)
│ ├── Pages/ or Components/
│ └── Program.cs
```
**Initial architecture from Axel**
```
┌─────────────────────────────────────────────────────────────────┐
│ INGESTION LAYER │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ SFTP Gateway │ │ API Gateway │ │
│ │ (Blob Storage) │ │ (Container App) │ │
│ └────────┬─────────┘ └────────┬─────────┘ │
│ │ │ │
└───────────┼──────────────────────────────────┼──────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ PROCESSING LAYER │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ Priority Queue │ │ Bulk Queue │ │
│ │ (Azure Queue) │ │ (Azure Queue) │ │
│ └────────┬────────┘ └────────┬─────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ File Processor │ │ File Processor │ │
│ │ (Single Docs) │ │ (Bulk Files) │ │
│ └────────┬────────┘ └────────┬─────────┘ │
│ │ │ │
│ └───────────┬─────────────┘ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Template Engine │ │
│ │ (Handlebars + PDF) │ │
│ └──────────┬───────────┘ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Channel Router │ │
│ └──────────┬───────────┘ │
│ │ │
└──────────────────────┼──────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ DELIVERY LAYER │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Postal │ │ Email │ │ Kivra │ │E-faktura│ ... │
│ │Exporter │ │ Sender │ │ Client │ │ Client │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```