Comment on page
Add Persister Configuration
The persister configuration is written in a YAML format. The INSERT and UPDATE queries for each table are added in prepared statement format, followed by the jsonPaths of values that have to be inserted/updated.
For example, for a table named studentinfo with id, name, age, and marks fields, the following configuration will get the persister ready to insert data into studentinfo table -
serviceMaps:
serviceName: student-management-service
mappings:
- version: 1.0
description: Persists student details in studentinfo table
fromTopic: save-student-info
isTransaction: true
queryMaps:
- query: INSERT INTO studentinfo( id, name, age, marks) VALUES (?, ?, ?, ?);
basePath: Students.*
jsonMaps:
- jsonPath: $.Students.*.id
- jsonPath: $.Students.*.name
- jsonPath: $.Students.*.age
- jsonPath: $.Students.*.marks
For our guide, for adding persister config, the following steps need to be followed -
Steps: i) Fork the configs repo. Ignore if done already. Clone configs repo in the local environment.
git clone -o upstream https://github.com/<your_configs_repo>/configs
ii) Persister configurations for all modules are present under the
egov-persister folder. Create a file by the name of btr-persister.yml (or any other name) under the egov-persister folder.iii) Add the following content to it -
serviceMaps:
serviceName: btr-services
mappings:
- version: 1.0
description: Persists birth details in tables
fromTopic: save-bt-application
isTransaction: true
queryMaps:
- query: INSERT INTO eg_bt_registration(id,tenantid,applicationnumber,babyfirstname,babylastname,fatherid,motherid,doctorname,hospitalname,placeofbirth,timeofbirth,createdby,lastmodifiedby,createdtime, lastmodifiedtime) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?,?,?);
basePath: BirthRegistrationApplications.*
jsonMaps:
- jsonPath: $.BirthRegistrationApplications.*.id
- jsonPath: $.BirthRegistrationApplications.*.tenantId
- jsonPath: $.BirthRegistrationApplications.*.applicationNumber
- jsonPath: $.BirthRegistrationApplications.*.babyFirstName
- jsonPath: $.BirthRegistrationApplications.*.babyLastName
- jsonPath: $.BirthRegistrationApplications.*.fatherOfApplicant.id
- jsonPath: $.BirthRegistrationApplications.*.motherOfApplicant.id
- jsonPath: $.BirthRegistrationApplications.*.doctorName
- jsonPath: $.BirthRegistrationApplications.*.hospitalName
- jsonPath: $.BirthRegistrationApplications.*.placeOfBirth
- jsonPath: $.BirthRegistrationApplications.*.timeOfBirth
- jsonPath: $.BirthRegistrationApplications.*.auditDetails.createdBy
- jsonPath: $.BirthRegistrationApplications.*.auditDetails.lastModifiedBy
- jsonPath: $.BirthRegistrationApplications.*.auditDetails.createdTime
- jsonPath: $.BirthRegistrationApplications.*.auditDetails.lastModifiedTime
- query: INSERT INTO eg_bt_address(id, tenantid, doorno, latitude, longitude, buildingname, addressid, addressnumber, type, addressline1, addressline2, landmark, street, city, locality, pincode, detail, registrationid, createdby, lastmodifiedby, createdtime, lastmodifiedtime) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
basePath: BirthRegistrationApplications.*
jsonMaps:
- jsonPath: $.BirthRegistrationApplications.*.address.id
- jsonPath: $.BirthRegistrationApplications.*.address.tenantId
- jsonPath: $.BirthRegistrationApplications.*.address.doorNo
- jsonPath: $.BirthRegistrationApplications.*.address.latitude
- jsonPath: $.BirthRegistrationApplications.*.address.longitude
- jsonPath: $.BirthRegistrationApplications.*.address.buildingName
- jsonPath: $.BirthRegistrationApplications.*.address.addressId
- jsonPath: $.BirthRegistrationApplications.*.address.addressNumber
- jsonPath: $.BirthRegistrationApplications.*.address.type
- jsonPath: $.BirthRegistrationApplications.*.address.addressLine1
- jsonPath: $.BirthRegistrationApplications.*.address.addressLine2
- jsonPath: $.BirthRegistrationApplications.*.address.landmark
- jsonPath: $.BirthRegistrationApplications.*.address.street
- jsonPath: $.BirthRegistrationApplications.*.address.city
- jsonPath: $.BirthRegistrationApplications.*.address.locality.name
- jsonPath: $.BirthRegistrationApplications.*.address.pincode
- jsonPath: $.BirthRegistrationApplications.*.address.detail
- jsonPath: $.BirthRegistrationApplications.*.id
- jsonPath: $.BirthRegistrationApplications.*.address.createdBy
- jsonPath: $.BirthRegistrationApplications.*.address.lastModifiedBy
- jsonPath: $.BirthRegistrationApplications.*.address.createdTime
- jsonPath: $.BirthRegistrationApplications.*.address.lastModifiedTime
- version: 1.0
description: Update birth registration applications in table
fromTopic: update-bt-application
isTransaction: true
queryMaps:
- query: UPDATE eg_bt_registration SET tenantid = ?,babyFirstName = ?, timeOfBirth = ? WHERE id=?;
basePath: BirthRegistrationApplications.*
jsonMaps:
- jsonPath: $.BirthRegistrationApplications.*.tenantId
- jsonPath: $.BirthRegistrationApplications.*.babyFirstName
- jsonPath: $.BirthRegistrationApplications.*.timeOfBirth
- jsonPath: $.BirthRegistrationApplications.*.id
Import all the core-services projects as Maven projects into your IDE. It is assumed that you have already cloned the DIGIT code locally.
Modify the application.properties file in the egov-persister project and set the following property:
egov.persist.yml.repo.path=file:///Users/subha/Code/configs/egov-persister/btr-persister.yml
Note: You can set a comma-separated list of files as the value of the above property. If you are running multiple services locally, then this has to be a comma-separated list of persister config files. Make sure you always give the absolute path.
Make sure the Spring DB configurations and Flyway config reflect the same database as what has been set in the module itself. Otherwise, we will see failures in the persister code.
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://localhost:5432/birthregistration3
spring.datasource.username=yourusername
spring.datasource.password=yourpassword
Make sure the Kafka is running locally. Now, go ahead and run the EgovPersistApplication from the IDE. Check the console to make sure it is listening to the right topics as configured in your module's application.properties file.
The persister is now ready for use.
Note: Below steps are for when you deploy your code to the DIGIT env, not for local development. You may choose to do this when you build and deploy.
Push the code to the appropriate branch from which your environment will read it.
v) Go to your fork of the DIGIT-DevOps repository. Under the
deploy-as-code/helm/environments directory, find the deployment helm chart that was used to deploy DIGIT. In the deployment helm chart (which was used to set up the DIGIT environment), find "egov-persister". Find the "persist-yml-path" property and add the path to your new persister file here.
In the snippet below,
file:///work-dir/configs/egov-persister/birth-module-developer-guide.yml1
egov-persister:
2
replicas: 6
3
persist-yml-path: "file:///work-dir/configs/egov-persister/privacy-audit.yml,file:///work-dir/configs/egov-persister/pgr-migration-batch.yml,file:///work-dir/configs/egov-persister/pgr-services-persister.yml,file:///work-dir/configs/egov-persister/pdf-filestoreid-update.yml,file:///work-dir/configs/egov-persister/chatbot.yml,file:///work-dir/configs/egov-persister/pt-mutation-calculator-persister.yml,file:///work-dir/configs/egov-persister/apportion-persister.yml,file:///work-dir/configs/egov-persister/property-services-registry.yml,file:///work-dir/configs/egov-persister/billing-services-persist.yml,file:///work-dir/configs/egov-persister/egf-bill.yaml,file:///work-dir/configs/egov-persister/egov-user-event-persister.yml,file:///work-dir/configs/egov-persister/egov-workflow-v2-persister.yml,file:///work-dir/configs/egov-persister/firenoc_persiter.yaml,file:///work-dir/configs/egov-persister/hrms-employee-persister.yml,file:///work-dir/configs/egov-persister/pdf-generator.yml,file:///work-dir/configs/egov-persister/pg-service-persister.yml,file:///work-dir/configs/egov-persister/pgr.v3.yml,file:///work-dir/configs/egov-persister/property-services.yml,file:///work-dir/configs/egov-persister/pt-calculator-v2-persister.yml,file:///work-dir/configs/egov-persister/pt-drafts.yml,file:///work-dir/configs/egov-persister/pt-persist.yml,file:///work-dir/configs/egov-persister/tl-billing-slab-persister.yml,file:///work-dir/configs/egov-persister/tl-calculation-persister.yml,file:///work-dir/configs/egov-persister/tradelicense.yml,file:///work-dir/configs/egov-persister/uploader-persister.yml,file:///work-dir/configs/egov-persister/collection-migration-persister.yml,file:///work-dir/configs/egov-persister/water-persist.yml,file:///work-dir/configs/egov-persister/water-meter.yml,file:///work-dir/configs/egov-persister/assessment-persister.yml,file:///work-dir/configs/egov-persister/sewerage-persist.yml,file:///work-dir/configs/egov-persister/bpa-persister.yml,file:///work-dir/configs/egov-persister/property-services-migration-temp-config.yml,file:///work-dir/configs/egov-persister/assessment-persister-migration-temp.yml,file:///work-dir/configs/egov-persister/migration-batch-count-persister.yml,file:///work-dir/configs/egov-persister/land-persister.yml,file:///work-dir/configs/egov-persister/noc-persister.yml,file:///work-dir/configs/egov-persister/fsm-persister.yaml,file:///work-dir/configs/egov-persister/vehicle-persister.yaml,file:///work-dir/configs/egov-persister/vendor-persister.yaml,file:///work-dir/configs/egov-persister/fsm-calculator-persister.yaml,file:///work-dir/configs/egov-persister/echallan.yml,file:///work-dir/configs/egov-persister/egov-document-upload-persister.yml,file:///work-dir/configs/egov-persister/egov-survey-service-persister.yml,file:///work-dir/configs/egov-persister/firenoc-calculator-persister.yml,file:///work-dir/configs/egov-persister/bulk-bill-generation-audit.yml,file:///work-dir/configs/egov-persister/nss-persister.yml,file:///work-dir/configs/egov-persister/birth-death.yml,file:///work-dir/configs/egov-persister/bulk-bill-generator-ws.yml,file:///work-dir/configs/egov-persister/bulk-bill-generator-sw.yml,file:///work-dir/configs/egov-persister/audit-service-persister.yml,file:///work-dir/configs/egov-persister/birth-module-developer-guide.yml"
4
vi) Raise a PR to the appropriate branch of the DevOps repo (master, in egov case) which was forked/used to create the deployment. Once that is merged, restart the indexer service in your environment so it will pick up this new config for the module.
Last modified 6mo ago