Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the SinglePipeDefinition Syntax #487

Merged
merged 4 commits into from
Dec 18, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/docs/docs/dev/04-guides/02-working-with-the-ast.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ assert(referenced !== undefined);
## AST wrapper classes

The generated interfaces for AST nodes in `ast.ts` are only meant to represent the AST structurally, they don't define any behavior.
Also, in case of syntactic sugar, there may be different kinds of AST nodes representing the same semantic language concept (e.g. single pipes with a verbose syntax or chained pipes).
Also, in case of syntactic sugar, there may be different kinds of AST nodes representing the same semantic language concept.

To cope with this problem, there is the concept of an `AstNodeWrapper`.
An AST node wrapper is capable of wrapping AST nodes that represent the same semantic language concept and adding behavior to them via custom methods.
Expand Down
22 changes: 0 additions & 22 deletions apps/docs/docs/user/core-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,28 +24,6 @@ pipeline CarsPipeline {
}
```

Alternatively, you can use a slightly longer syntax for pipes:

```jayvee
pipeline CarsPipeline {
// Assumption: blocks "GasReserveHttpExtractor", "GasReserveCSVInterpreter", "GasReserveTableInterpreter", and "GasReserveLoader" are defined

pipe {
from: GasReserveHttpExtractor;
to: GasReserveTextFileInterpreter;

}

pipe {
from: GasReserveTextFileInterpreter;
to: GasReserveCSVInterpreter;

}

// etc.
}
```

## Blocks

A `Block` is a processing step within a `Pipeline`.
Expand Down
41 changes: 18 additions & 23 deletions example/cars.jv
Original file line number Diff line number Diff line change
Expand Up @@ -16,64 +16,59 @@ pipeline CarsPipeline {
// usually at the top of the pipeline.
// by connecting blocks via pipes.

// 3. Verbose syntax of a pipe
// 3. Syntax of a pipe
// connecting the block CarsExtractor
// with the block CarsTextFileInterpreter.
pipe {
from: CarsExtractor;
to: CarsTextFileInterpreter;
}

// 4. The output of the "from" block is hereby used
// as input for the "to" block.
CarsExtractor -> CarsTextFileInterpreter;

// 5. More convenient syntax of a pipe
CarsTextFileInterpreter -> CarsCSVInterpreter;
// 4. The output of the preceding block is hereby used
// as input for the succeeding block.

// 6. Pipes can be further chained,
// 5. Pipes can be further chained,
// leading to an overview of the pipeline.
CarsCSVInterpreter
CarsTextFileInterpreter
-> CarsCSVInterpreter
-> NameHeaderWriter
-> CarsTableInterpreter
-> CarsLoader;


// 7. Below the pipes, we usually define the blocks
// 6. Below the pipes, we usually define the blocks
// that are connected by the pipes.

// 8. Blocks instantiate a blocktype by using the oftype keyword.
// 7. Blocks instantiate a blocktype by using the oftype keyword.
// The blocktype defines the available properties that the block
// can use to specify the intended behavior of the block
block CarsExtractor oftype HttpExtractor {

// 9. Properties are assigned to concrete values.
// 8. Properties are assigned to concrete values.
// Here, we specify the URL where the file shall be downloaded from.
url: "https://gist.githubusercontent.com/noamross/e5d3e859aa0c794be10b/raw/b999fb4425b54c63cab088c0ce2c0d6ce961a563/cars.csv";
}

// 10. The HttpExtractor requires no input and produces a binary file as output.
// 9. The HttpExtractor requires no input and produces a binary file as output.
// This file has to be interpreted, e.g., as text file.
block CarsTextFileInterpreter oftype TextFileInterpreter { }

// 11. Next, we interpret the text file as sheet.
// 10. Next, we interpret the text file as sheet.
// A sheet only contains text cells and is useful for manipulating the shape of data before assigning more strict value types to cells.
block CarsCSVInterpreter oftype CSVInterpreter {
enclosing: '"';
}

// 12. We can write into cells of a sheet using the CellWriter blocktype.
// 11. We can write into cells of a sheet using the CellWriter blocktype.
block NameHeaderWriter oftype CellWriter {
// 13. We utilize a syntax similar to spreadsheet programs.
// 12. We utilize a syntax similar to spreadsheet programs.
// Cell ranges can be described using the keywords "cell", "row", "column", or "range" that indicate which
// cells are selected for the write action.
at: cell A1;

// 14. For each cell we selected with the "at" property above,
// 13. For each cell we selected with the "at" property above,
// we can specify what value shall be written into the cell.
write: ["name"];
}

// 15. As a next step, we interpret the sheet as a table by adding structure.
// 14. As a next step, we interpret the sheet as a table by adding structure.
// We define a valuetype per column that specifies the data type of the column.
// Rows that include values that are not valid according to the their valuetypes are dropped automatically.
block CarsTableInterpreter oftype TableInterpreter {
Expand All @@ -94,7 +89,7 @@ pipeline CarsPipeline {
];
}

// 16. As a last step, we load the table into a sink,
// 15. As a last step, we load the table into a sink,
// here into a sqlite file.
// The structural information of the table is used
// to generate the correct table.
Expand All @@ -103,7 +98,7 @@ pipeline CarsPipeline {
file: "./cars.sqlite";
}

// 17. Congratulations!
// 16. Congratulations!
// You can now use the sink for your data analysis, app,
// or whatever you want to do with the cleaned data.
}
9 changes: 0 additions & 9 deletions libs/language-server/src/grammar/pipeline.langium
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,4 @@ PipelineDefinition:
'}';

PipeDefinition:
SinglePipeDefinition | ChainedPipeDefinition;

SinglePipeDefinition:
'pipe' '{'
'from' ':' from=[BlockDefinition] ';'
'to' ':' to=[BlockDefinition] ';'
'}';

ChainedPipeDefinition:
blocks+=[BlockDefinition] ('->' blocks+=[BlockDefinition])+ ';';
13 changes: 8 additions & 5 deletions libs/language-server/src/lib/ast/model-util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ import {
} from './generated/ast';
// eslint-disable-next-line import/no-cycle
import { BlockTypeWrapper, ConstraintTypeWrapper } from './wrappers';
import { PipeWrapper, createSemanticPipes } from './wrappers/pipe-wrapper';
import {
PipeWrapper,
createWrappersFromPipeChain,
} from './wrappers/pipe-wrapper';

export function collectStartingBlocks(
container: PipelineDefinition | CompositeBlocktypeDefinition,
Expand Down Expand Up @@ -90,12 +93,12 @@ function collectPipes(
const pipeline = block.$container;
const allPipes = collectAllPipes(pipeline);

return allPipes.filter((semanticPipe) => {
return allPipes.filter((pipeWrapper) => {
switch (kind) {
case 'outgoing':
return semanticPipe.from === block;
return pipeWrapper.from === block;
case 'ingoing':
return semanticPipe.to === block;
return pipeWrapper.to === block;
case undefined:
return false;
}
Expand All @@ -108,7 +111,7 @@ export function collectAllPipes(
): PipeWrapper[] {
const result: PipeWrapper[] = [];
for (const pipe of container.pipes) {
result.push(...createSemanticPipes(pipe));
result.push(...createWrappersFromPipeChain(pipe));
}
return result;
}
Expand Down
112 changes: 25 additions & 87 deletions libs/language-server/src/lib/ast/wrappers/pipe-wrapper.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,102 +9,56 @@ import { DiagnosticInfo } from 'langium';
import {
BlockDefinition,
BlocktypePipeline,
ChainedPipeDefinition,
PipeDefinition,
SinglePipeDefinition,
isSinglePipeDefinition,
} from '../generated/ast';

import { AstNodeWrapper } from './ast-node-wrapper';

export class PipeWrapper<N extends PipeDefinition = PipeDefinition>
implements AstNodeWrapper<N>
export class PipeWrapper<T extends PipeDefinition | BlocktypePipeline>
implements AstNodeWrapper<T>
{
public readonly astNode: N;
private readonly chainIndex?: number;
public readonly astNode: T;
private readonly chainIndex: number;
public readonly from: BlockDefinition;
public readonly to: BlockDefinition;

constructor(
pipe: ChainedPipeDefinition | BlocktypePipeline,
chainIndex: number,
);
constructor(pipe: SinglePipeDefinition);
constructor(pipe: N, chainIndex?: number) {
constructor(pipe: T, chainIndex: number) {
this.astNode = pipe;
if (isSinglePipeDefinition(pipe)) {
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
assert(pipe.from?.ref !== undefined);
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
assert(pipe.to?.ref !== undefined);
this.from = pipe.from.ref;
this.to = pipe.to.ref;
} else {
assert(chainIndex !== undefined);
assert(0 <= chainIndex && chainIndex + 1 < pipe.blocks.length);
assert(pipe.blocks[chainIndex]?.ref !== undefined);
assert(pipe.blocks[chainIndex + 1]?.ref !== undefined);
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
this.from = pipe.blocks[chainIndex]!.ref!;
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
this.to = pipe.blocks[chainIndex + 1]!.ref!;
this.chainIndex = chainIndex;
}
assert(0 <= chainIndex && chainIndex + 1 < pipe.blocks.length);
assert(pipe.blocks[chainIndex]?.ref !== undefined);
assert(pipe.blocks[chainIndex + 1]?.ref !== undefined);
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
this.from = pipe.blocks[chainIndex]!.ref!;
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
this.to = pipe.blocks[chainIndex + 1]!.ref!;
this.chainIndex = chainIndex;
}

getFromDiagnostic(): DiagnosticInfo<PipeDefinition> {
if (isSinglePipeDefinition(this.astNode)) {
const result: DiagnosticInfo<SinglePipeDefinition> = {
node: this.astNode,
property: 'from',
};
return result;
}
assert(this.chainIndex !== undefined);
const result: DiagnosticInfo<ChainedPipeDefinition> = {
getFromDiagnostic(): DiagnosticInfo<T, keyof T> {
return {
node: this.astNode,
property: 'blocks',
index: this.chainIndex,
};
return result;
}

getToDiagnostic(): DiagnosticInfo<PipeDefinition> {
if (isSinglePipeDefinition(this.astNode)) {
const result: DiagnosticInfo<SinglePipeDefinition> = {
node: this.astNode,
property: 'to',
};
return result;
}
assert(this.chainIndex !== undefined);
const result: DiagnosticInfo<ChainedPipeDefinition> = {
getToDiagnostic(): DiagnosticInfo<T, keyof T> {
return {
node: this.astNode,
property: 'blocks',
index: this.chainIndex + 1,
};
return result;
}

equals(pipe: PipeWrapper): boolean {
equals(pipe: PipeWrapper<T>): boolean {
return this.from === pipe.from && this.to === pipe.to;
}

static canBeWrapped(
pipe: ChainedPipeDefinition | BlocktypePipeline,
pipe: PipeDefinition | BlocktypePipeline,
chainIndex: number,
): boolean;
static canBeWrapped(pipe: SinglePipeDefinition): boolean;
static canBeWrapped<N extends PipeDefinition>(
pipe: N,
chainIndex?: number,
): boolean {
if (isSinglePipeDefinition(pipe)) {
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
return pipe.from?.ref !== undefined && pipe.to?.ref !== undefined;
}
return (
chainIndex !== undefined &&
0 <= chainIndex &&
chainIndex + 1 < pipe.blocks.length &&
pipe.blocks[chainIndex]?.ref !== undefined &&
Expand All @@ -113,32 +67,16 @@ export class PipeWrapper<N extends PipeDefinition = PipeDefinition>
}
}

export function createSemanticPipes(
pipe: PipeDefinition | BlocktypePipeline,
): PipeWrapper[] {
if (isSinglePipeDefinition(pipe)) {
return createFromSinglePipe(pipe);
}
return createFromChainedPipe(pipe);
}

function createFromSinglePipe(pipe: SinglePipeDefinition): PipeWrapper[] {
if (PipeWrapper.canBeWrapped(pipe)) {
return [new PipeWrapper(pipe)];
}
return [];
}

function createFromChainedPipe(
pipe: ChainedPipeDefinition | BlocktypePipeline,
): PipeWrapper[] {
const result: PipeWrapper[] = [];
export function createWrappersFromPipeChain<
A extends PipeDefinition | BlocktypePipeline,
>(pipe: A): PipeWrapper<A>[] {
const result: PipeWrapper<A>[] = [];
for (let chainIndex = 0; chainIndex < pipe.blocks.length - 1; ++chainIndex) {
if (!PipeWrapper.canBeWrapped(pipe, chainIndex)) {
continue;
}
const semanticPipe = new PipeWrapper(pipe, chainIndex);
result.push(semanticPipe);
const pipeWrapper = new PipeWrapper(pipe, chainIndex);
result.push(pipeWrapper);
}
return result;
}
Loading
Loading