After setting up Elasticsearch locally I needed to seed it with some dummy data.
I came across this article that explains all the steps in details, including populating the postgres with dummy data to begin with.
I did not want to populate the dummy data since I already had some data in postgres.
But next step, to export the data into json format proved difficult because I use dockerized postgres.
> export.json would not work for me, because
psql ran with pager, and would wait at
-- More -- prompt. I saw a first couple of lines in
-- More -- at the end.
I tried some options like turning the pager off from
psqlrc - which works locally, but I could not get it to work in docker.
Then I decided to break it into separate steps.
First, get the data into json format:
docker exec psql and in interactive mode ran the following:
Now I had
export.json inside the docker container. So
docker exec -it my-pgsql bash
I found the file in root folder, so I moved it to location that is accessible from outside.
cp /export.json /var/lib/postgresql/data/
Now I got it out using
docker cp command
docker cp <container_id>:/var/lib/postgresql/data/export.json ./raw_export.json
See this on how to get the correct volume path
I had to format it so that elasticsearch would accept it
cat raw_export.json | sed 's/^\[//' | sed 's/^[[:blank:]]//' | sed 's/[[:blank:]]$//' | sed 's/,$//' | sed 's/]$//' > export.json
Rest of the steps were similar to the original article. I had to change
format-json.js per my database structure.
The original article is from late 2018. Both elasticsearch and elasticdump have changed since then.
I had problem with
https for local elasticsearch server. I bypassed it via setting up the environment variable
NODE_TLS_REJECT_UNAUTHORIZED=0 node_modules/elasticdump/bin/elasticdump --input=formatted.json --output=https://dev:elastic_dev@localhost:9200/
See the documentation here
That was it.
I was able to search via Kibana 🎉