Ryan Jarvinen Open Shift Talk @ Postgres Open 2013
1 like•2,176 views
The document discusses writing portable PostgreSQL applications for the open cloud. It covers open cloud platforms like OpenStack that avoid lock-in, and platforms like OpenShift that provide portable PostgreSQL applications using cartridges and scaling hooks. Examples are given of simple applications like a TODO app using Flask, SQLAlchemy and PostgreSQL that can be easily deployed to OpenShift.
Ryan Jarvinen Open Shift Talk @ Postgres Open 2013
- 1. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 1/48 PG OPEN CLOUD Portable PG applications http://bit.ly/1eddml6 http://ryanjarvinen.com/presentations/pg_open_cloud
- 2. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 2/48 presented by /ryan jarvinen @ryanj
- 3. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 3/48 Open Platforms Engineer at Red Hat ryanj@redhat.com
- 4. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 4/48 AGENDA 1. Open Cloud Overview 2. Learn how to write for the Open Cloud
- 5. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 5/48 the Cloud "what is it made of?"
- 6. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 6/48 The cloud is: hot air? a series of tubes? mostly cat photos✓
- 7. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 7/48 INFRASTRUCTURE Communications: fiber, switches, wireless Storage and Compute: chips, servers, disks, databases … DevOps (and friction) … Software as a Service: Dropbox, Gmail
- 8. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 8/48 IAAS Amazon EC2 RackSpace Linode BlueHost * Many are VM / VPS-based services
- 9. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 9/48 BUILDING WITH POSTGRES NORMALLY LOOKS LIKE:
- 10. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 10/48 1. Install Apache with Mod_WSGI 2. Make sure we have all the Python extensions set so Apache can see them 3. Configure firewall so that only http, https, and ssh are available to outside world 4. Install Postgresql, and plugins 5. update PG_hba.conf, create db user, create db user password, set up access privs, and a test account 6. Create DB, give user permissions to the DB 7. Set up a git repository and then figure out how to structure and deploy 8. Now I can write code - hello world, then database connection, then real app 9. Get it working with a public DNS 10. Now you are stuck maintaining the machine
- 11. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 11/48 OR - if your Ops team offers pre-built VMs: 1. Fill out a mile high stack of request forms for a VM from Ops and wait 2. Then realize it's not configured properly, submit ticket and wait 3. Repeat same information, and wait 4. Your ops person moves to a different group, GOTO Line 1....
- 12. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 12/48 As a sysadmin: 1. We're busy keeping the site up, we'll need to provision your VM later 2. Developers always want to try a new language or datastore that they don't know how to install, let alone manage 3. There is whole bunch of account creation and lockdown for each VM. I'll need to give them Sudo or they'll annoy me all the time 4. Each machine ends up with a wildly different config 5. Developers never give up the resources from their VMs - spreadsheets and emails 6. If there is a security fix I need to go into each VM and update it 7. I don't want to spend my time updating DNS tables
- 13. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 13/48 PLATFORMS PROVIDE A PEACEFUL ENVIRONMENT FOR DEVS AND SYSADMINS Operations can ensure stability and performance Developers can self-provision environments without waiting The discussion shifts toward establishing policies for scaling
- 14. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 14/48
- 15. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 15/48 PUBLIC / HOSTED PAAS Heroku EngineYard Google App Engine Amazon BeanStalk * usually VM-based or proprietary application container technologies
- 16. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 16/48 the public cloud
- 17. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 17/48 This all sounds great, but what about… LOCK-IN?
- 18. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 18/48 THE OPEN CLOUD
- 19. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 19/48 IAAS OpenStack Eucalptus CloudStack Or, roll your own
- 20. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 20/48
- 21. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 21/48 OPEN PAAS OpenShift Cloud Foundry Built around Containers
- 22. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 22/48 TRIUMPH OF THE OPEN CLOUD When "Big Cloud" fails you, you can always decide to host your own
- 23. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 23/48 TINYTINYRSS Reusable, Redeployable code - host it anywhere https://github.com/openshift-quickstart/tiny_tiny_rss- openshift-quickstart rhc app create rss php-5 postgresql-9 cron-1.4 --from-code=https:/ /github.com/openshift-quickstart/tiny_tiny_rss-openshift-quickstar t.git https://rss-shifter.rhcloud.com/
- 24. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 24/48 ADDING POSTGRES TO EXISTING APPLICATIONS: done! rhc cartridge add postgres-8.4 rhc cartridge add postgres-9.2 blog post: PostgreSQL 9.2 Comes to OpenShift
- 25. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 25/48 Or, while connected over ssh: rhc app show rss env | grep DB OPENSHIFT_POSTGRESQL_DB_PASSWORD=lXcFVx4hIZgR OPENSHIFT_POSTGRESQL_DB_SOCKET=/var/lib/openshift/523672f7e0b8cd02 d70003bc/postgresql/socket/ OPENSHIFT_POSTGRESQL_DB_HOST=127.7.8.130 OPENSHIFT_POSTGRESQL_DB_PID=/var/lib/openshift/523672f7e0b8cd02d70 003bc/postgresql/pid/postgres.pid OPENSHIFT_POSTGRESQL_DB_USERNAME=adminpahue6e OPENSHIFT_POSTGRESQL_DB_URL=postgresql://adminpahue6e:lXcFVx4hIZgR @127.7.8.130:5432 OPENSHIFT_POSTGRESQL_DB_PORT=5432 OPENSHIFT_POSTGRESQL_DB_LOG_DIR=/var/lib/openshift/523672f7e0b8cd0 2d70003bc/postgresql/log/
- 26. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 26/48 TERMINOLOGY (RED HAT) Broker – Management host, orchestration of Nodes Node – Compute host containing Gears Gear – Allocation of fixed memory, compute, and storage resources for running applications Cartridge – A technology/framework (Python, Ruby, Javascript, PHP, Perl, Java/JEE, PG, MySQL, etc.) for application service dependencies
- 27. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 27/48 GEARS: SECURE AND EFFICIENT ,SELinux Cgroups
- 28. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 28/48 AN OPEN CARTRIDGE FORMAT cart developer's guide
- 29. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 29/48
- 30. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 30/48 OPEN, AND WORKING IN THE UPSTREAM
- 31. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 31/48 WRITING APPS FOR THE OPEN CLOUD
- 32. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 32/48 TODO APP (FLASK AND SQLALCHEMY) https://github.com/shekhargulati/todo-flask-openshift- quickstart blog post: Build Your App on OpenShift Using Flask, SQLAlchemy, and PostgreSQL 9.2
- 33. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 33/48 rhc app create -s task python-2.7 postgresql-9.2 --from-code=https ://github.com/shekhargulati/todo-flask-openshift-quickstart.git http://task-shifter.rhcloud.com/
- 34. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 34/48 OPENSHIFT SCALING HOOKS Setting a min and max scale from the command line rhc cartridge-scale python-2.7 -a task --min 3 --max 5
- 35. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 35/48 SCALING WEB UI
- 36. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 36/48 HAPROXY SCALING HOOKS Adding and removing instances / scaling up and down HAProxy on / off switches These hooks are available on the command line within your gear, and via a REST API haproxy_ctld --up haproxy_ctld --down haproxy_ctld_daemon start haproxy_ctld_daemon stop haproxy_ctld_daemon restart
- 37. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 37/48 HAPROXY WEB UI http://$YOUR_APP_DNS/haproxy-status/ HAPROXY RAW DATA http://$YOUR_APP_DNS/haproxy-status/;csv
- 38. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 38/48 LIVING IN THE CLOUDS Code for America
- 39. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 39/48 ADOPT A CAUSE This app should be using PostGIS for spatial queries: https://github.com/openshift-quickstart/adopt-a- hydrant-openshift-quickstart blog post: More Spatial in Your Database with PostGIS 2.0
- 40. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 40/48 Persist configuration details, while keeping your source clean: rhc app create hydrant ruby-1.9 postgresql-8.4 --from=code=http:// github.com/ryanj/adopt-a-hydrant.git --env SECRET_TOKEN="YOUR_SECR ET_TOKEN" http://hydrant-shifter.rhcloud.com/
- 41. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 41/48 ENVIRONMENT VARIABLES Listing your custom env vars: Setting a variable: Help with configuration: cd hydrant rhc env list rhc env set SECRET_TOKEN="a1fdacc3b1d14d6a92ed1219ed304d02529f5350 85262a90c39f072ef6de0ee9fe3a3d0194f02a2a8eb3" rhc help env
- 42. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 42/48 DEPENDENCIES Automatic support for dependency resolution using standard packaging, native to each language: (ruby), (python), and (node.js)gems eggs npm modules
- 43. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 43/48 ACTION HOOKS 1. enable postgis 2. bootstrap your db https://github.com/ryanj/pg_parks/tree/master/.openshift/action_ho
- 44. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 44/48 BANNERS? http://citibike-shifter.rhcloud.com/ rhc env set CITIBIKE_NEARBY_STATIONS="268,257,258" -a citibikes
- 45. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 45/48 DISTRIBUTED MULTI-CLOUD FEDERATED APPLICATIONS
- 46. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 46/48 JOIN THE COMMUNITY Meritocracy for our pull requests Help with anything from core, quickstarts, and cartridges, to small typo fixes in the command line tools PEPs for major feature enhancements Contribution Guidelines Public Trello cards Public Test plans Public Bugzilla Vote on Features
- 47. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 47/48 Check out the upstream source: Try our hosted solution (3 apps free): Request an evaluation for: OpenShift Origin OpenShift Online OpenShift Enterprise
- 48. 9/21/13 Writing portable postgreSQL-backed applications for the Open Cloud ryanjarvinen.com/presentations/pg_open_cloud/#/ 48/48 Thanks for following along! --ryanj slides: http://bit.ly/1eddml6