Unlocking The Value Of Data Across The Organization Through User Friendly Data Tools With Prophecy
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the. . .
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here. Support the show!23 May 2022Unlocking The Value Of Data Across The Organization Through User Friendly Data Tools With Prophecy - E292Rewind 10 seconds1XSkip 30 seconds ahead0:00/0:00Share on social media:DescriptionTranscriptChaptersSummaryThe interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. With an eye to making data workflows more accessible to everyone in an organization Raj Bains and his team at Prophecy designed a powerful and extensible low-code platform that lets technical and non-technical users scale data flows without forcing everyone into the same layers of abstraction. In this episode he explores the tension between code-first and no-code utilities and how he is working to balance the strengths without falling prey to their shortcomings. AnnouncementsHello and welcome to the Data Engineering Podcast, the show about modern data managementWhen you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go todataengineeringpodcast. com/linodetoday and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go todataengineeringpodcast. com/atlantoday and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscriptionSo now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today atdataengineeringpodcast. com/selectstar. You’ll also get a swag package when you continue on a paid plan. Your host is Tobias Macey and today I’m interviewing Raj Bains about how improving the user experience for data tools can make your work as a data engineer better and easierInterviewIntroductionHow did you get involved in the area of data management?What are the broad categories of data tool designs that are available currently and how does that impact what is possible with them?What are the points of friction that are introduced by the tools?Can you share some of the types of workarounds or wasted effort that are made necessary by those design elements?What are the core design principles that you have built into Prophecy to address these shortcomings?How do those user experience changes improve the quality and speed of work for data engineers?How has the Prophecy platform changed since we last spoke almost a year ago?What are the tradeoffs of low code systems for productivity vs. flexibility and creativity?What are the most interesting, innovative, or unexpected approaches to developer experience that you have seen for data tools?What are the most interesting, unexpected, or challenging lessons that you have learned while working on user experience optimization for data tooling at Prophecy?When is it more important to optimize for computational efficiency over developer productivity?What do you have planned for the future of Prophecy?Contact InfoLinkedIn@_raj_bainson TwitterParting QuestionFrom your perspective, what is the biggest gap in the tooling or technology for data management today?Closing AnnouncementsThank you for listening! Don’t forget to check out our other show,Podcast. __init__to learn about the Python language, its community, and the innovative ways it is being used. Visit thesiteto subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Emailhosts@dataengineeringpodcast. com) with your story. To help other people find the show please leave a review oniTunesand tell your friends and co-workersLinksProphecyPodcast EpisodeCUDAClustrixHortonworksApache HiveCompilerworksPodcast EpisodeAirflowDatabricksFivetranPodcast EpisodeAirbytePodcast EpisodeStreamsetsChange Data CaptureApache PigSparkScalaAb InitioType 2 Slowly Changing DimensionsAWS DeequMatillionPodcast EpisodeProphecy SaaSThe intro and outro music is fromThe HugbyThe Freak Fandango Orchestra/CC BY-SASupport Data Engineering PodcastShare on social media:Listen in your favorite app:PodversePodcast GuruApple PodcastsPocket CastsOvercastSpotifyPodlinkMore optionsHere are shows you might likeSee show recommendationsAI Engineering PodcastTobias MaceyThe Python Podcast. __init__Tobias Macey