Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
D
DataCatalog
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Container registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
eFlows4HPC WP2
DataCatalog
Commits
592fe88c
Commit
592fe88c
authored
3 years ago
by
Jedrzej Rybicki
Browse files
Options
Downloads
Patches
Plain Diff
arch update
parent
700e114f
No related branches found
No related tags found
No related merge requests found
Pipeline
#73191
failed
3 years ago
Stage: test
Stage: build
Stage: deploy
Changes
1
Pipelines
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
arch/arch.adoc
+21
-9
21 additions, 9 deletions
arch/arch.adoc
with
21 additions
and
9 deletions
arch/arch.adoc
+
21
−
9
View file @
592fe88c
...
...
@@ -12,7 +12,7 @@ include::src/config.adoc[]
[[section-introduction-and-goals]]
== Introduction and Goals
Following describes the architecture of eF
L
ows4HPC Data Catalog. The service
Following describes the architecture of eF
l
ows4HPC Data Catalog. The service
will provide information about data sets used in the project. The catalog will
store info about locations, schemas, and additional metadata.
...
...
@@ -54,9 +54,8 @@ Main features:
|===
| **Constraint** | **Explanation**
| Authentication |
There is no solution for that in the project yet, local authenticator?
| Authentication |
OAuth-based for admin users
| Deployment | We shall use CI/CD, the project will also be a playing field to setup this and test before the Data Logistics
| github vs. gitlab | Not sure if we can use our local gitlab
| Docker-based Deployment | This technology will be used in the project anyways
|===
...
...
@@ -84,11 +83,24 @@ Admin -> Data Catalog: either a web page or CLI
== Solution Strategy
=== Speed and flexibility
This product will not be very mission critical, we want to keep it simple. A solution even without a backend database would be possible.
Probably we
will use some noSQL database for maximal flexibility.
API with Swagger/OpenAPI (e.g. fastAPI). Frontend static page with JavaScript calls to the API.
This product will not be very mission critical, we want to keep it simple. A solution even without a backend database would be possible.
API with Swagger/OpenAPI (e.g. fastAPI). Frontend static page with JavaScript calls to the API.
=== Deployment
with Jenkins
===
Automatic
Deployment
1. Jenkins instance in HDF Cloud to setup Pipelines,
2. Code in Github/Gitlab
3. Automatic deployment with Docker? Docker-compose
1. Code in Gitlab
2. Resources on HDF Cloud
3. Automatic deployment with Docker + docker-compose, OpenStack API
We use docker image repository in gitlab to generate new images.
=== Structure
Main data model is based on JSON and uses pydantic. Resources in the Catalog are of two storage classes (sources and targets). The number of classes
can change in the future.
The actual storage of the information in the catalog is done through an abstract interface which in the first attempt stores the data in a file, other
backends can be added.
API uses a backend abstraction to mange the informations
Web front-end are static html files generated from templates. This gives a lot of flexibility and allows for easy scalability if required.
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment