tag:blogger.com,1999:blog-65548161743659097372024-03-19T04:48:47.521-04:00Thoughts on Product ManagementLessons after 20+ years in the product trenchesRichard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.comBlogger16125tag:blogger.com,1999:blog-6554816174365909737.post-78340889270373510692022-01-03T15:12:00.006-05:002022-01-04T11:04:08.996-05:00The #1 Mistake Made by Product People at All Levels<p><span data-offset-key="foo-0-0"><span data-text="true">In my 20+ year career in product management for B2B enterprise companies, I have seen product managers at every level make a certain kind of mistake. It is so easy to make that I occasionally make it myself when I'm not careful.</span></span></p><div data-draftjs-conductor-fragment="{"blocks":[{"key":"fpb6g","text":"In my 20+ year career in product management for B2B enterprise companies, I have seen product managers at every level make a certain kind of mistake. It is so easy to make that I occasionally make it myself when I'm not careful.\n","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"6n27d","text":"What is this mistake? It is to spend too much time on tasks and deliverables that are not core to the product function, which is to to determine and define products to be built. If you keep falling into this trap then ultimately you can't be effective at your job and your company won't sell compelling products.\n","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"a369a","text":"Your primary job as a product manager is to figure out what your market and customers need and make sure it gets built. If you aren't careful, you can spend all of your time performing tasks in support of products such as sales enablement, customer success, product marketing, and pre-sales.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"><div class="_25Ehb _3qYRK Zn7O0 public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr rich_content_P" data-block="true" data-editor="editor" data-offset-key="endcb-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="endcb-0-0"><span data-offset-key="endcb-0-0"><span data-text="true">What is this mistake? It is to spend too much time on tasks and deliverables that are not core to the product function, which is to to determine and define products to be built. If you keep falling into this trap then ultimately you can't be effective at your job and your company won't sell compelling products.
</span></span></div></div><div class="_25Ehb _3qYRK Zn7O0 public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr rich_content_P" data-block="true" data-editor="editor" data-offset-key="2hs0o-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="2hs0o-0-0"><span data-offset-key="2hs0o-0-0"><span data-text="true">Your primary job as a product manager is to figure out what your market and customers need and make sure it gets built. If you aren't careful, you can spend all of your time performing tasks in support of products such as sales enablement, customer success, product marketing, and pre-sales.</span></span></div><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="2hs0o-0-0"><div data-draftjs-conductor-fragment="{"blocks":[{"key":"5ek3a","text":"How Do You Know This Is Happening?","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"><h3 style="text-align: left;">How Do You Know This Is Happening?</h3><div style="text-align: left;"><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"><div class="_25Ehb _3qYRK Zn7O0 public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr rich_content_P" data-block="true" data-editor="editor" data-offset-key="7viv1-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="7viv1-0-0"><span data-offset-key="7viv1-0-0"><span data-text="true">It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:</span></span></div></div><ul class="public-DraftStyleDefault-ul" data-offset-key="fnmgs-0-0"><li class="MIezR Zn7O0 _2zLWO public-DraftStyleDefault-list-ltr public-DraftStyleDefault-unorderedListItem public-DraftStyleDefault-reset public-DraftStyleDefault-depth0 public-DraftStyleDefault-listLTR" data-block="true" data-editor="editor" data-offset-key="fnmgs-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="fnmgs-0-0"><span data-offset-key="fnmgs-0-0"><span data-text="true">Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.</span></span></div></li><li class="MIezR Zn7O0 _2zLWO public-DraftStyleDefault-list-ltr public-DraftStyleDefault-unorderedListItem public-DraftStyleDefault-depth0 public-DraftStyleDefault-listLTR" data-block="true" data-editor="editor" data-offset-key="boade-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="boade-0-0"><span data-offset-key="boade-0-0"><span data-text="true">Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.</span></span></div></li><li class="MIezR Zn7O0 _2zLWO public-DraftStyleDefault-list-ltr public-DraftStyleDefault-unorderedListItem public-DraftStyleDefault-depth0 public-DraftStyleDefault-listLTR" data-block="true" data-editor="editor" data-offset-key="326fq-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="326fq-0-0"><span data-offset-key="326fq-0-0"><span data-text="true">The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.</span></span></div></li><li class="MIezR Zn7O0 _2zLWO public-DraftStyleDefault-list-ltr public-DraftStyleDefault-unorderedListItem public-DraftStyleDefault-depth0 public-DraftStyleDefault-listLTR" data-block="true" data-editor="editor" data-offset-key="akgtd-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="akgtd-0-0"><span data-offset-key="akgtd-0-0"><span data-text="true">Customer Success needs a "quick" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.</span></span></div></li><li class="MIezR Zn7O0 _2zLWO public-DraftStyleDefault-list-ltr public-DraftStyleDefault-unorderedListItem public-DraftStyleDefault-depth0 public-DraftStyleDefault-listLTR" data-block="true" data-editor="editor" data-offset-key="34aor-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="34aor-0-0"><span data-offset-key="34aor-0-0"><span data-text="true">Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.</span></span></div></li></ul><div class="_25Ehb _3qYRK Zn7O0 public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr rich_content_P" data-block="true" data-editor="editor" data-offset-key="aqa5m-0-0"><div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="aqa5m-0-0"><span data-offset-key="aqa5m-0-0"></span></div></div>All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful. <br /><h3 style="text-align: left;">Consequences of the Problem</h3>The problem, however, is when you are so busy fulfilling urgent requests like this that you have no time for the strategic job of product management, which is to determine what you should be building. Understanding the market requires time to talk to customers and understand the pain points they're trying to solve, or time to study competitors and their documentation to determine how you can uniquely differentiate yourself in the market.</div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"></div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"></div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"> </div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;">It is easy to feel like you are adding value to the organization by performing lots of tasks that seem high priority and are valued by other teams. But if you can't spend time on strategy or customer and market research then ultimately you aren't being effective at your job. Your company will execute well on the products it has but in the long run it won't be competitive and uniquely differentiated.</div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"></div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"><h3 style="text-align: left;">How Do You Respond?</h3></div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;"></div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="white-space: pre-wrap;">So how do you avoid this problem? As with many things, the first step is to realize when this is happening. If your to-do list is filled with accomplishing tasks or deliverables for other teams and you aren't dedicating blocks of time to talking to customers about their pain points or researching competitors then you are most likely getting bogged down.
<p style="-qt-block-indent: 0; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; margin: 0px; text-indent: 0px;">How you respond is dependent on your role within the organization. If you are an individual contributor such as a product manager or senior product manager, let your manager know that you are getting bogged down in tasks that aren't contributing to your core mission as a product manager. Give concrete examples of what you are asked to do and how much time this is taking. Work with your manager to gently push back on other teams that are requesting your time on all of these tasks.
</p>
<p style="-qt-block-indent: 0; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; margin: 0px; text-indent: 0px;">If you manage product managers or own the product function at your organization and this problem is either happening to you or your direct reports, it is up to you to solve it. One key is to clarify the tasks that Product is responsible for, versus other functions like Customer Success or Product Marketing. </p><p style="-qt-block-indent: 0; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; margin: 0px; text-indent: 0px;"> </p><p style="-qt-block-indent: 0; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; margin: 0px; text-indent: 0px;">Here are some examples:</p><ul style="text-align: left;"><li>Who owns RFPs at your company? Product wants to contribute to the success of the company by helping with RFPs but another group such as Product Marketing or Customer Success typically owns them. This other group needs to take responsibility for incorporating Product's responses into RFP boilerplates and using less of Product's time.</li><li>Who owns sales enablement? This is typically handled by Product Marketing or a similar team. This team needs to get more up to speed on the product so they can regularly handle tasks like creating bullets describing the product's unique value proposition. Product can help but this isn't part of their core job.</li><li>Who owns ensuring that docs are technically accurate? This typically isn't owned by any one group but it shouldn't fall all to Product. Engineering should get involved. If no one else is reviewing docs then this should be escalated to a manager or director within Engineering.</li><li>Who owns the beta process? Running a beta involves an extensive amount of project management and would ideally be handled by a dedicated project manager. Product may manage some projects as part of the feature development process but running a beta shouldn't be done by Product just because they are good at managing projects. Usually someone within Engineering should own the beta process but explicitly make this decision rather than having Product do it by default. </li></ul></div><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="text-align: left; white-space: pre-wrap;"><div style="-qt-block-indent: 0; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; margin: 0px; text-align: left; text-indent: 0px;"><h3 style="text-align: left;">The Importance of Ownership </h3></div><p style="-qt-block-indent: 0; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; margin: 0px; text-indent: 0px;">Note that I use the word "own" a lot in these examples. This is key to figuring out how to solve this problem. As a rule of thumb: </p></div><ul style="text-align: left;"><li>Spend more time on functions owned by Product like roadmap planning and market or customer analysis.</li><li>Spend less time on functions owned by other teams. If you are spending too much time here then go to the owner of the function and clarify to them that you are glad to help but this isn't part of your core job and you need to spend less time.</li></ul><div data-draftjs-conductor-fragment="{"blocks":[{"key":"7416e","text":"It is easy to fall into this trap for many reasons. Here are a few scenarios that come to mind:","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"irrc","text":"Product Marketing is working on an RFP for one of the biggest deals of the quarter and there's no one else available who can answer detailed questions about your product, especially in answering about features that your product doesn't support yet.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"f85g","text":"Sales needs help putting together a set of bullets describing your product's unique value proposition and how it solves problems better than the competitors.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"c4kcu","text":"The Docs team needs someone to review the docs of the latest feature or release and they can't seem to get anyone in Engineering to take the time to look them over for correctness.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"arl7d","text":"Customer Success needs a \"quick\" doc to explain a complicated new feature to a handful of premier customers who want to implement it quickly.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"dim05","text":"Customer Success and Engineering want to launch a beta of the upcoming release but they need someone who can lead the beta program, organizing a list of trial customers, coming up with a test plan of new features and coordinating weekly calls with each customer.","type":"unordered-list-item","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"ca8j0","text":"","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"3ds5a","text":"All of these scenarios are perfectly valid and move the business forward. I've spent considerable time myself on helping with these types of requests. Especially in a startup, we need to do whatever we can to make the business successful.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"8sq96","text":"","type":"header-three","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":{}}],"entityMap":{},"VERSION":"8.63.7"}" style="text-align: left; white-space: pre-wrap;"><p style="-qt-block-indent: 0; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; margin: 0px; text-indent: 0px;"> </p></div></div></div><span data-offset-key="2hs0o-0-0"><span data-text="true"></span></span></div></div></div>Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com1tag:blogger.com,1999:blog-6554816174365909737.post-91387785997346593732014-11-29T23:36:00.004-05:002016-07-29T21:17:24.645-04:00Creating a HackintoshI've always wanted to create a "Hackintosh", i.e. a standard PC that runs OS X. My PC is over 5 years old so it was time for a refresh. I figured this was the best time to give the Hackintosh a go.<br />
<br />
<h3>
<b>Hardware</b></h3>
<div>
<b><br /></b></div>
<ul>
<li><b>CPU:</b> Intel Quad Core i7 4790 3.6 Ghz</li>
<li><b>Motherboard:</b> GIGABYTE GA-Z97-HD3</li>
<ul>
<li><b>Audio:</b> ALC 887</li>
<li><b>Network: </b>Realtek 8111F-VL</li>
</ul>
<li><b>Network Card:</b> 4 Antennas 802.11ac WiFi BCM94360CD Wireless Network Card</li>
<li><b>Graphics Card:</b> nVidia 750 GTX</li>
<li><b>Memory:</b> Corsair Vengeance DDR3-1600 32 GB (4x8 GB)</li>
<li><b>Hard Drive</b>: Seagate ST3000DM001 3 TB SATA3 7200 rpm</li>
<li><b>DVD:</b> Samsung SH-224DB 24X</li>
</ul>
<ul>
</ul>
<h3>
<b><br /></b></h3>
<h3>
<b>BIOS Changes</b></h3>
<div>
<br /></div>
<div>
The first step was to change the BIOS settings to support OS X. Disabling VT-d is the only setting that is clearly required; the others are questionable but were done by others so I thought they were worth trying.</div>
<div>
<br /></div>
<div>
<b>F7</b> to load Optimized Defaults</div>
<div>
<ul>
<li><b>M.I.T.</b></li>
<ul>
<li><b>Advanced Frequency Settings</b></li>
<ul>
<li><b>Extreme Memory Profile (X.M.P.): </b>Enabled</li>
</ul>
<li><b>Miscellaneous Settings</b></li>
<ul>
<li><b>PCIe Slot Configuration:</b> Gen3</li>
</ul>
</ul>
<li><b>BIOS Features</b></li>
<ul>
<li><b>Intel Virtualization Technology:</b> Disabled</li>
<li><b>VT-d</b>: Disabled</li>
</ul>
<li><b>Peripherals</b></li>
<ul>
<li><b>XHCI Mode: </b>Auto</li>
<li><b>XHCI Hand-off: </b>Enabled</li>
<li><b>EHCI Hand-off:</b> Enabled</li>
</ul>
</ul>
<h3>
<b><br /></b></h3>
<h3>
<b>Installing Clover</b></h3>
</div>
<div>
<br /></div>
<div>
Initially I installed <a href="http://www.tonymacx86.com/445-unibeast-install-os-x-yosemite-any-supported-intel-based-pc.html" target="_blank">Unibeast onto a USB drive</a> to run the OS X Installer but couldn't get past some errors. In retrospect, the errors were caused by the 750 GTX card.</div>
<div>
<br /></div>
<div>
I decided to <a href="http://www.tonymacx86.com/yosemite-desktop-guides/144426-how-install-os-x-yosemite-using-clover.html" target="_blank">install Clover</a> because it would give me more control over the installation process. I followed Steps 1 and 2 to download OS X Yosemite and create the bootable USB drive. I won't repeat the steps here because I followed them exactly. I chose RealtekRTL81xx.kext as the network card driver. I chose config.plist-Standard.zip as my minimal configuration.</div>
<div>
<br /></div>
<div>
When I booted the Clover USB drive I added these settings to the boot options:</div>
<div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">-v nv_disable=1</span></div>
<div>
<br /></div>
<div>
This had the effect of disabling the video card and going into verbose mode so I could see the boot messages. Clover will hang and reboot unless you disable the Nvidia driver.</div>
<div>
<br /></div>
<div>
I then followed the instructions in Step 3 to use Disk Utility to create a partition on the hard drive then install OS X Yosemite on the drive.</div>
<div>
<br /></div>
<div>
When the install finished, it restarted the machine. Since the next step automatically runs stage 2 of the install process (create the recovery partition and complete the install), it was important to again add "nv_disable=1" to the boot options during this restart. Otherwise the boot would fail and the installer would not go to stage 2, requiring me to start over.</div>
<div>
<br /></div>
<h3>
<b>Post-Installation</b></h3>
<div>
<br /></div>
<div>
I followed Step 4 in reformatting the EFI partition to FAT32 and installing Clover on the hard drive so it could boot without the USB drive. I added RealtekRTL81xx.kext and FakeSMC.kext to /Volumes/EFI/EFI/Clover/kexts/10.10.</div>
<div>
<br /></div>
<div>
I also followed the instructions in running Clover Configurator and imessage_debug.zip to generate a new config.plist to install in /Volumes/EFI/EFI/Clover.</div>
<div>
<br /></div>
<div>
Note that I removed nv_disable=1 and added nvda_drv=1 to the boot options because of the next step.</div>
<div>
<br /></div>
<h3>
<b>Installing Video and Audio Drivers</b></h3>
<div>
<br /></div>
<div>
I installed this <a href="http://www.tonymacx86.com/graphics/143435-nvidia-releases-alternate-graphics-drivers-10-10-0-343-01-01-maxwell-geforce-gtx-970-980-support.html" target="_blank">Nvidia experimental driver</a> to get the 750 GTX working. Note that it only works with 10.10.0; I had previously installed 10.10.1 and had to start over all again to get this driver installed. Once you install this driver then you must activate it with the nvda_drv=1 boot option.</div>
<div>
<br /></div>
<div>
There is now an <a href="http://www.tonymacx86.com/graphics/149031-nvidia-releases-alternate-graphics-drivers-10-10-1-343-01-02-a-3.html" target="_blank">Nvidia driver for 10.10.1</a> but I'm not sure if I'm ready to move to it yet. There are reports that the DVI port doesn't work yet, which is what I use. This <a href="http://www.tonymacx86.com/general-hardware-discussion/125264-nvidia-launches-maxwell-geforce-gtx-750-750-ti-graphics-cards-15.html#post927939" target="_blank">comment</a> may come in handy.</div>
<div>
<br /></div>
<div>
To get sound working, I used this <a href="http://www.tonymacx86.com/audio/143757-audio-realtek-alc-applehda-guide.html" target="_blank">audio guide</a> to point me to the right place. I wound up pursuing the cloverALC/Clover option. I downloaded <a href="https://github.com/toleda/audio_CloverALC/blob/master/audio_cloverALC-100.command.zip">audio_cloverALC-100.command.zip</a> from <a href="https://github.com/toleda/audio_CloverALC" target="_blank">this site</a>. After unzipping the file, I ran it and answered "Yes" to all the questions.<br />
<br />
To get the network card working, I downloaded the .zip file from <a href="https://github.com/RehabMan/OS-X-Realtek-Network" target="_blank">RehabMan's fork</a> of Mieze's Realtek RTL8111 Network Driver.</div>
Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-79039175750220095612013-02-20T23:19:00.000-05:002013-02-20T23:20:31.422-05:00Using chef to build out a Hadoop clusterAfter not posting for a while, I have about 3-4 posts that I'd like to get out there.
The first is about using chef to build a Hadoop cluster.<br />
<br />
<a href="http://www.opscode.com/chef/">Chef</a> is a configuration management tool that allows one to automate the process of provisioning servers. I had to create a Hadoop cluster of 4-5 servers and I wanted to use this opportunity to automate the process with chef.<br />
<br />
I had to perform a series of the same steps on these Linux nodes:<br />
<ul>
<li>Install ruby and chef</li>
<li>Install Sun Java</li>
<li>Install VMware Tools</li>
<li>Install NTP</li>
<li>Add its hostname to a shared /etc/hosts file</li>
<li>Configure passwordless ssh login</li>
</ul>
<h4>
Installing Chef and Ruby</h4>
I followed the steps in <a href="http://wiki.opscode.com/display/chef/Fast+Start+Guide">this link</a>.<br />
<br />
The first step is to sign up for a Hosted Chef account on the <a href="http://www.opscode.com/">Opscode</a> site. An account is free for 5 nodes or less. Perform the following steps:<br />
<ol>
<li>Create a new organization</li>
<li>Select "Generate knife config" to download knife.rb</li>
<li>Select "Regenerate validation key" to download (validator).pem</li>
</ol>
<br />
<li>Click on your account and click "get private key" to download (private key).pem</li>
Then install ruby and chef on your first host. Once you do the first host, you can quickly bootstrap the others.
Install Ruby:<br />
<pre class="brush: ruby" name="code">sudo apt-get update
sudo apt-get install ruby ruby-dev libopenssl-ruby rdoc ri irb build-essential wget ssl-cert git-core
</pre>
Install rubygems:<br />
<pre class="brush: ruby" name="code">cd /tmp
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.10.tgz
tar zxf rubygems-1.8.10.tgz
cd rubygems-1.8.10
sudo ruby setup.rb --no-format-executable
</pre>
Install chef:<br />
<pre class="brush: ruby" name="code">sudo gem install chef
cd ~
git clone https://github.com/opscode/chef-repo.git
mkdir -p ~/chef-repo/.chef
cp (private key).pem ~/chef-repo/.chef
cp (validator).pem ~/chef-repo/.chef
cp knife.rb ~/chef-repo/.chef
</pre>
Connect to Hosted Chef and configure workstation as a client:<br />
<pre class="brush: ruby" name="code">cd ~/chef-repo
knife configure client ./client-config
sudo mkdir /etc/chef
sudo cp -r ~/chef-repo/client-config/* /etc/chef
sudo chef-client
</pre>
Once the client is installed on the first host, you can bootstrap the clients on the other hosts by using this command, as described <a href="http://wiki.opscode.com/display/chef/Client+Bootstrap+Fast+Start+Guide">here</a>.
Bootstrap the other clients. This assumes you have created a user called hadoop who is the main hadoop user.<br />
<pre class="brush: ruby" name="code">knife bootstrap (node IP) -x hadoop -P (password) --sudo
</pre>
Repeat this for all of your other chef nodes.
<br />
<h4>
Installing some Chef recipes</h4>
Now that chef is installed on all the nodes, it's time to run some chef recipes. A recipe is a set of configuration instructions. In my case, I want to install some packages. I started with VMware Tools, Sun Java, and NTP.
<br />
Start by creating a new cookbook:
<br />
<pre class="brush: ruby" name="code">knife cookbook create MYCOOKBOOK
</pre>
Then download some existing cookbooks from the <a href="http://wiki.opscode.com/display/chef/Chef+Repository">Chef Repository</a>.
<br />
<pre class="brush: ruby" name="code">knife cookbook site install vmtools
knife cookbook site install java
knife cookbook site install ntp
</pre>
Add these recipes to each node's run list:
<br />
<pre class="brush: ruby" name="code">knife node run_list add NODE_NAME "recipe[java:sun]"
knife node run_list add NODE_NAME "recipe[vmtools]"
knife node run_list add NODE_NAME "recipe[ntp]"
</pre>
You'll then need to run "sudo chef-client" on each node to execute the run list and install these packages.
<br />
<h4>
Populate /etc/hosts</h4>
The next step is to create a recipe that will populate the /etc/hosts file from the Chef repository. One of Hadoop's requirements is to store the name-IP mapping for every node in the cluster in /etc/hosts. The easiest way to do this is to populate /etc/hosts from the list of hosts that Chef knows about.
<br />
So start by creating your new recipe in your cookbook. I call it "hosts":
<br />
<pre class="brush: ruby" name="code">knife cookbook create hosts
</pre>
Your cookbook will now have a subdirectory called hosts with some skeleton files already created. Create your default ruby script in hosts/recipes/default.rb:
<br />
<pre class="brush: ruby" name="code"># Gets list of names from all nodes in repository and rewrites /etc/hosts
hosts = {}
localhost = nil
search(:node, "name:*", %w(ipaddress fqdn)) do |n|
hosts[n["ipaddress"]] = n
end
template "/etc/hosts" do
source "hosts.erb"
mode 0644
variables(:hosts => hosts)
end
</pre>
Now edit the hosts.erb file, stored in hosts/templates/default/hosts.erb:
<br />
<pre class="brush: ruby" name="code">hosts/templates/default/hosts.erb:
127.0.0.1 localhost
<% @hosts.keys.sort.each do |ip| %>
<%= ip %> <%= @hosts[ip]["fqdn"] %>
<% end %>
</pre>
Now deploy it to all your hosts:
<br />
<pre class="brush: ruby" name="code">knife node run_list add NODE_NAME "recipe[hosts]"
</pre>
Don't forget to run "sudo chef-client" on each node.<br />
You should also upload this recipe to the Chef server:
<br />
<pre class="brush: ruby" name="code">knife cookbook upload hosts
</pre>
<h4>
Installing passwordless ssh login</h4>
A Hadoop cluster requires passwordless ssh login between the master and its slave nodes. The easiest way to do this is to have each node create its own SSH keys with an empty password, and then copy the public keys for all nodes to the master node.
<br />
<br />
So create a recipe to create the SSH key with empty password. I call it "sshlogin":
<br />
<pre class="brush: ruby" name="code">knife cookbook create sshlogin
</pre>
Create your default ruby script in sshlogin/recipes/default.rb:
<br />
<pre class="brush: ruby" name="code">
# Create empty RSA password
execute "ssh-keygen" do
command "sudo -u hadoop ssh-keygen -q -t rsa -N '' -f /home/hadoop/.ssh/id_rsa"
creates "/home/hadoop/.ssh/id_rsa"
action :run
end
# Copy public key to node1; if key doesn't exist in authorized_keys, append it to this file
execute <<EOF
cat /home/hadoop/.ssh/id_rsa.pub | sudo -u hadoop ssh hadoop@node1 "(cat > /tmp/tmp.pubkey; mkdir -p .ssh; touch .ssh/authorized_keys; grep #{node[:fqdn]} .ssh/authorized_keys > /dev/null || cat /tmp/tmp.pubkey >> .ssh/authorized_keys; rm /tmp/tmp.pubkey)"
EOF
</pre>
Note that when you run this recipe on each host, it will prompt you to type the password of node1 each time because you are essentially scp'ing the key to this master node.
<br />
Now you can deploy this recipe:
<pre class="brush: ruby" name="code">
knife cookbook upload sshlogin
knife node run_list add node2 "recipe[sshlogin]"
</pre>
Type this command to run the recipes on each host:
<pre class="brush: ruby" name="code">
sudo chef-client
</pre>Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com1tag:blogger.com,1999:blog-6554816174365909737.post-80482793936426574292011-05-16T23:28:00.000-04:002011-05-16T23:28:33.003-04:00Building a Hadoop clusterI've recently had to build a Hadoop cluster for a <a href="http://www.apl.jhu.edu/~paulmac/ir.html">class in information retrieval</a>. My final project involved building a Hadoop cluster.<br />
<br />
Here are some of my notes on configuring the nodes in the cluster.<br />
<br />
These links on configuring a <a href="http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/">single node</a> cluster and <a href="http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/">multi node</a> cluster were the most helpful.<br />
<br />
I downloaded the latest Hadoop distribution then moved it into /hadoop. I had problems with this latest distribution (v.21) so I used v.20 instead.<br />
<br />
Here are the configuration files I changed:<br />
<br />
core-site.xml:<br />
<pre class="brush: ruby" name="code"> <property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
</pre><br />
hadoop-env.sh:<br />
<pre class="brush: ruby" name="code"># Variables required by Mahout
export HADOOP_HOME=/hadoop
export HADOOP_CONF_DIR=/hadoop/conf
export MAHOUT_HOME=/Users/rpark/mahout
PATH=/hadoop/bin:/Users/rpark/mahout/bin:$PATH
# The java implementation to use. Required.
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
</pre><br />
hdfs-site.xml:<br />
<pre class="brush: ruby" name="code"> <property>
<name>dfs.replication</name>
<value>3</value>
</property>
</pre><br />
mapred-site.xml:<br />
<pre class="brush: ruby" name="code"> <property>
<name>mapred.job.tracker</name>
<value>master:9001</value>
</property>
</pre><br />
masters:<br />
<pre class="brush: ruby" name="code">master
</pre><br />
slaves:<br />
<pre class="brush: ruby" name="code">master
slave1
slave2
slave3
slave4
</pre><br />
Be sure to enable password-less ssh between master and slaves. Use this command to create an SSH key with an empty password:<br />
<pre class="brush: ruby" name="code">ssh-keygen -t rsa -P ""
</pre><br />
Enable password-less ssh login for the master to itself:<br />
<pre class="brush: ruby" name="code">cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
</pre><br />
Then copy id_rsa.pub to each slave and do the same with each slave's authorized_keys file.<br />
<br />
I ran into a few errors along the way. Here is an error that gave me a lot of trouble in the datanode logs:<br />
<pre class="brush: ruby" name="code">2011-05-08 01:04:30,032 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_1804860059826635300_1001 received exception org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block blk_1804860059826635300_1001 is valid, and cannot be written to.
</pre><br />
The solution was to use hostnames every time I referenced a host, either itself or a remote host. I set a host's own name in /etc/hostname and the others in /etc/hosts. I used these hostnames in /hadoop/conf/masters, slaves, and the various conf files.<br />
<br />
Every so often I ran into this error in the datanode logs:<br />
<pre class="brush: ruby" name="code">... ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 308967713; datanode namespaceID = 113030094
at org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:281)
at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:121)
at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:230)
at org.apache.hadoop.dfs.DataNode.(DataNode.java:199)
at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:1202)
at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1146)
at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:1167)
at org.apache.hadoop.dfs.DataNode.main(DataNode.java:1326)
</pre><br />
I fixed this by deleting tmp/dfs/data on the datanodes where I saw the error. Unfortunately, I had to reformat the HDFS volume after I did this.<br />
<br />
I had to raise the ulimit for open files. On Ubuntu nodes I edited /etc/security/limits.conf:<br />
<pre class="brush: ruby" name="code">rpark soft nofile 8192
rpark hard nofile 8192
</pre><br />
For OS X nodes I just edited ~/.profile:<br />
<pre class="brush: ruby" name="code">ulimit -n 8192
</pre><br />
I ran into this error when copying data into HDFS:<br />
<pre class="brush: ruby" name="code">could only be replicated to 0 nodes, instead of 1
</pre><br />
The solution was simply to wait for the datanode to start up. I usually saw the error when I immediately copied data into HDFS after starting the cluster.<br />
<br />
Port 50070 on the namenode gave me a Web UI to tell me how many nodes were in the cluster. This was very useful.Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com1tag:blogger.com,1999:blog-6554816174365909737.post-36383635615345109932011-05-16T11:23:00.000-04:002011-05-16T22:27:36.275-04:00Working with VMware vShield REST API in perlHere is an overview of how to use perl code to work with VMware's vShield API.<br />
<br />
vShield App and Edge are two security products offered by VMware. vShield Edge has a broad range of functionality such as firewall, VPN, load balancing, NAT, and DHCP. vShield App is a NIC-level firewall for virtual machines.<br />
<br />
We'll focus today on how to use the API to programatically make firewall rule changes. Here are some of the things you can do with the API:<br />
<ul><li>List the current firewall ruleset</li>
<li>Add new rules</li>
<li>Get a list of past firewall revisions</li>
<li>Revert back to a previous ruleset revision</li>
</ul>vShield API documentation is available <a href="http://www.vmware.com/pdf/vshield_41_api.pdf">here</a>.<br />
<br />
Before we get into the API itself, let's look at what the firewall ruleset looks like. It's formatted as XML:<br />
<pre class="brush: ruby" name="code"><?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<vshieldzonesfirewallconfiguration>
<containerassociation>
<container id="1.1.1.1/32"><ipaddress>1.1.1.1/32</IPAddress></Container>
<container id="10.1.1.1/32"><ipaddress>10.1.1.1/32</IPAddress></Container>
<container id="My Datacenter"><instanceid>datacenter-2</InstanceId></Container>
<container id="ANY"><name>ANY</Name></Container>
</ContainerAssociation>
<ruleset>
<rule>
<id>1023</ID>
<precedence>High</Precedence>
<position>1</Position>
<source ref="1.1.1.1/32" exclude="false"/>
<destination ref="10.1.1.1/32" exclude="false"/>
<sourceports>ANY</SourcePorts><
Application type="UNICAST">LDAP over SSL</Application>
<destinationports>636</DestinationPorts>
<protocol>TCP</Protocol>
<action>ALLOW</Action>
<log>deny</Log>
<notes></Notes>
</Rule>
<rule>
<id>1020</ID>
<precedence>Low</Precedence>
<position>3</Position>
<source ref="My Datacenter" exclude="false"/>
<destination ref="My Datacenter" exclude="true"/>
<sourceports>ANY</SourcePorts>
<application type="UNICAST">IMAP</Application>
<destinationports>143</DestinationPorts>
<protocol>TCP</Protocol><
Action>ALLOW</Action>
<log>false</Log>
<Notes/>
</Rule>
</RuleSet>
</vshieldZonesFirewallConfiguration>
</pre>Here are some notes about the XML configuration:<br />
<ul><li>The API works mainly with container objects. A container can range from a datacenter or cluster all the way down to a port group or IP address.</li>
<li>Every container object must be listed in the <containerassociation> section.</li>
<li>Container objects have instance IDs. The instance ID is also referred to as the managed object ID (MOID)</li>
<li>Every firewall rule has its own ID as well as precedence and position fields.</li>
</ul><br />
If you want to edit a firewall ruleset, you must specify which ruleset you want. Every object has its own ruleset. So you could edit a ruleset at the datacenter level, cluster level, etc. all the way down to the port group level.<br />
<br />
For simplicity let's work with the ruleset at the datacenter level because this will cover all VMs in that datacenter.<br />
<br />
The first thing to do is get the object ID for the datacenter. If you don't already know this then you must look it up. There are two places you can find it:<br />
<ol><li>Use the Managed Object Browser in vCenter Server, located at https://<vcenter IP>/mob, e.g., https://10.1.1.1/mob</li>
<li>Query your vCenter Server with the vSphere SDK</li>
</ol><br />
Either way you must have access to vCenter Server.<br />
<br />
Here is some perl code to query vCenter. The code assumes that $dc_name is set to the name of your datacenter. You can find this name in the vSphere client.<br />
<pre class="brush: ruby" name="code">use VMware::RunTime;
$ENV{'VI_SERVER'} = $vc_ip;
$ENV{'VI_USERNAME'} = $vc_user;
$ENV{'VI_PASSWORD'} = $vc_pass;
# read/validate options and connect to the server
Opts::add_options(%opts);
Opts::parse();
Opts::validate();
Util::connect();
$view = Vim::find_entity_views(view_type => 'Datacenter');
foreach $datacenter (@$view) {
if (lc($datacenter->{name}) eq lc($dc_name)) {
return $datacenter->{mo_ref}->{value};
}
}
return "Not_found";
</pre><br />
Note that this code requires you include the .pm files from the perl SDK. You can find these in lib/VMware/share/VMware/ in the perl SDK tarball.<br />
<br />
Once you have the object ID for your datacenter, you can use it to create the vShield URL that you will need to access the datacenter's firewall ruleset:<br />
<pre class="brush: ruby" name="code">$url = "https://" . $vsm_ip . "/api/1.0/zones/" . $moid . "/firewall/rules";
</pre>Note that this URL is to access the ruleset in vShield App. If you want to access the ruleset in vShield Edge instead, simply change the "zones" in the URL to "network". So the resulting vShield Edge URL looks like this:<br />
<pre class="brush: ruby" name="code">$url = "https://" . $vsm_ip . "/api/1.0/network/" . $moid . "/firewall/rules";
</pre>Now that you have the URL, use it to get the ruleset with a simple HTTP GET using Basic Authentication:<br />
<pre class="brush: ruby" name="code">$ua = LWP::UserAgent->new;
$request = HTTP::Request->new(GET =>$url);
$request->authorization_basic($vsm_user, $vsm_pass);
$response = $ua->request($request);
</pre><br />
$response now contains the XML ruleset. Copy it to a variable such as $ruleset and use your favorite XML library to work directly with each rule. I found that using XML::LibXML provides the best routines for both parsing and editing the XML.<br />
<br />
This code iterates through each rule, loading the source address and protocol into variables.<br />
<pre class="brush: ruby" name="code">my $parser = new XML::LibXML;
my $tree = $parser->parse_string($ruleset);
my $root = $tree->documentElement();
foreach my $rule_ref ($root->findnodes('RuleSet/Rule')) {
$rule_src = $rule_ref->findvalue('Source/@ref');
$rule_prot = $rule_ref->findvalue('Protocol');
}
</pre><br />
Note that the source address is accessible as an attribute named "ref" in the source tag. XPath syntax uses '@' to access XML attributes.<br />
<br />
The vShield API has certain restrictions when it comes to adding firewall rules. You can't just add a rule to the existing ruleset. Every time you update the ruleset with new rules, you replace all of the old rules.<br />
<br />
The proper way to add a rule is to load the existing rules into memory as an XML tree, add the new rules to the tree, then post the updated tree back as the new ruleset.<br />
<br />
This sample code illustrates how to add a new rule. Note that only a few of the fields are included here but every field in the rule is required, with the exception of Notes. You will get an error if you leave out a required field.<br />
<pre class="brush: ruby" name="code">my $rule_ref = XML::LibXML::Element->new("Rule");
my $id_el = XML::LibXML::Element->new("ID");
$id_el->appendText("0");
my $src_el = XML::LibXML::Element->new("Source");
$src_el->setAttribute("ref", $src_ip);
$src_el->setAttribute("exclude", "false");
$rule_ref->addChild($id_el);
$rule_ref->addChild($src_el);
my $rule_root = $root->findnodes('RuleSet')->get_node(1);
$rule_root->addChild($rule_ref);
</pre><br />
Here are some notes:<br />
<ul><li>To add a new rule, specify an ID of 0. When vShield adds the new rule to the ruleset, it will automatically generate a new ID.</li>
<li>The Position field is required but you can set it to any value. I set it to a default of 50. vShield Manager rewrites this field every time you move rules around in the vShield GUI.</li>
</ul>After you add the rule to the XML tree, you must also add a new container object for the IP addresses referenced by the rule:<br />
<pre class="brush: ruby" name="code">my $contain_root = $root->findnodes('ContainerAssociation')->get_node(1);
my $contain_el = XML::LibXML::Element->new("Container");
$contain_el->setAttribute("id", $ip_addr);
my $ip_addr_el = XML::LibXML::Element->new("IPAddress");
$ip_addr_el->appendText($ip_addr);
$contain_el->addChild($ip_addr_el);
$contain_root->addChild($contain_el);
</pre><br />
When you're done updating the XML tree, post the complete ruleset:<br />
<pre class="brush: ruby" name="code">$ua = LWP::UserAgent->new;
$request = HTTP::Request->new(POST=>$self->{url});
$request->authorization_basic($self->{vsm_user}, $self->{vsm_pass});
$request->content_type('application/xml');
$request->content($root->toString());
$response = $ua->request($request);
</pre>Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-59545479112304505482010-10-26T17:44:00.000-04:002012-01-18T16:40:06.955-05:00Using multiple versions of Ruby on the same hostI've recently come across a tool called <a href="http://rvm.beginrescueend.com/">RVM</a> or Ruby Version Manager. It enables you to run different versions of Ruby on the same host.<br />
<br />
RVM uses git so my first step was to <a href="http://wiki.github.com/mxcl/homebrew/installation">install git</a> with the Homebrew package manager. Homebrew is an increasingly popular alternative to MacPorts and Fink.<br />
<br />
Note that you'll need to install Xcode first.<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">/usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"<br />
brew install git</div><br />
Then I just followed the instructions available <a href="http://rvm.beginrescueend.com/rvm/install/">here</a>.<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )<br />
source ~/.rvm/scripts/rvm<br />
rvm install jruby,1.9.2-head </div><br />
Here is what the output looks like:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">info: Downloading jruby-bin-1.5.1, this may take a while depending on your connection...<br />
info: Extracting jruby-bin-1.5.1 ...<br />
info: Building Nailgun<br />
info: Installing JRuby to /Users/rpark/.rvm/rubies/jruby-1.5.1<br />
info: Importing initial gems...<br />
info: Installing rake<br />
info: Installing Ruby from source to: /Users/rpark/.rvm/rubies/ruby-1.9.2-head<br />
info: Downloading source from http://svn.ruby-lang.org/repos/ruby/branches/ruby_1_9_2.<br />
info: Copying from repo to src path...<br />
info: Running autoconf<br />
info: Configuring ruby-1.9.2-head, this may take a while depending on your cpu(s)...<br />
info: Compiling ruby-1.9.2-head, this may take a while depending on your cpu(s)...<br />
info: Installing ruby-1.9.2-head<br />
info: Installation of ruby-1.9.2-head is complete.<br />
info: Updating rubygems for /Users/rpark/.rvm/gems/ruby-1.9.2-head@global<br />
info: Updating rubygems for /Users/rpark/.rvm/gems/ruby-1.9.2-head<br />
info: adjusting shebangs for ruby-1.9.2-head (gem irb erb ri rdoc testrb rake).<br />
info: Importing initial gems...<br />
</div><br />
The install is done. When you want to switch between different versions of ruby, just use the "rvm use" command:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">rvm use 1.9.2-head<br />
11:10:23[~:15]$ which ruby<br />
/Users/rpark/.rvm/rubies/ruby-1.9.2-head/bin/ruby<br />
11:11:06[~:16]$ which gem<br />
/Users/rpark/.rvm/rubies/ruby-1.9.2-head/bin/gem<br />
rvm use jruby<br />
11:10:01[~:11]$ which ruby<br />
/Users/rpark/.rvm/rubies/jruby-1.5.1/bin/ruby<br />
11:10:06[~:12]$ which gem<br />
/Users/rpark/.rvm/rubies/jruby-1.5.1/bin/gem<br />
</div><br />
Then add this to the end of ~/.profile:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"<br />
</div><br />
RVM allows you to use <a href="http://rvm.beginrescueend.com/gemsets/basics/">different gemsets</a> with each ruby version. One use case is working with Rails 2.3.3 and Rails 3.0. I didn't need this feature but it's nice to know it's there.<br />
<br />
One final step is TextMate integration. This is described <a href="http://rvm.beginrescueend.com/integration/textmate/">here</a>.<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">rvm wrapper 1.9.2-head<br />
</div><br />
Then run this script:<br />
<br />
<pre class="brush: ruby" name="code">#!/usr/bin/env bash
mkdir -p /Library/Application\ Support/TextMate/
sudo chown -R $(whoami) /Library/Application\ Support/TextMate
cd /Library/Application\ Support/TextMate/
if [[ -d Bundles/.svn ]] ; then
cd Bundles && svn up
else
if [[ -d Bundles ]] ; then
mv Bundles Bundles.old
fi
svn co http://svn.textmate.org/trunk/Bundles
fi
exit 0
</pre><br />
Open up Shell Variables in TextMate's Preferences -> Advanced<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">Set TM_RUBY to /Users/rpark/.rvm/bin/textmate_ruby<br />
</div><br />
Final step:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">cd /Applications/TextMate.app/Contents/SharedSupport/Support/lib/ ; mv Builder.rb Builder.rb.backup<br />
</div>Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-54502021776956128082010-10-04T21:25:00.000-04:002010-10-04T21:25:10.531-04:00Connecting to JDBC data source from OS X perlHere's another blog post on a similar topic. I recently had to figure it out and I documented it already, so I thought I would share it here too so I don't forget.<br />
<br />
Here are instructions on connecting to a JDBC data source from OS X perl. I used <a href="http://search.cpan.org/~vizdom/DBD-JDBC-0.70/JDBC.pod">DBD::JDBC</a>.<br />
<br />
The process mainly involves setting up a local Java server that provides the front end for a JDBC driver. The Java code implements the JDBC connection to the data source. The perl code talks to the Java server on localhost via DBI so it can access the database.<br />
<br />
Here are the steps:<br />
<br />
1. Check out any documentation about required setup for your JDBC data source. For example, you may need to install an SSL certificate from the server to your client. You may also need to ensure that your server permits database access at all from your client or IP address.<br />
<br />
2. Download your JDBC driver and put any .jar files into a lib directory. For example, <a href="http://vjdbc.sourceforge.net/">VJDBC</a> is a JDBC driver that enables you to establish JDBC connections over Java RMI. It comes with vjdbc.jar.<br />
<br />
3. Install the perl modules Convert::BER, DBI, and DBD::JDBC. You can either install the modules from CPAN or install <a href="http://www.activestate.com/activeperl">ActiveState perl</a>. If you install from CPAN, you'll need to install Xcode first so you have a C compiler. If you use ActiveState, use PPM to install the required modules. Note that you'll still need to download the DBD::JDBC tarball from CPAN and install it manually because it doesn't show up in ppm.<br />
<br />
4. Copy dbd_jdbc.jar from the DBD::JDBC tarball into the same lib directory as the other .jar files you've collected.<br />
<br />
5. Download the latest version of log4j and copy log4j.jar into the same lib directory as above. You should now have at least 3 jar files: dbd_jdbc, log4j-1.2.16, vjdbc.<br />
<br />
6. Set your CLASSPATH variable to point to your jar files. If you want this variable to be always set, add it to your .profile.<br />
<br />
7. Startup the Java server with this command:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">java -Djdbc.drivers=com.sourcefire.vjdbc.VirtualDriver -Ddbd.port=9001 com.vizdom.dbd.jdbc.Server<br />
</div><br />
Note that I set the port to 9001. This is completely arbitrary and can be whatever you want it to be. But whatever you set this to, make sure you indicate this port in your perl code.<br />
<br />
You'll probably see something like this:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">log4j:WARN No appenders could be found for logger (com.vizdom.dbd.jdbc.Server).<br />
log4j:WARN Please initialize the log4j system properly.<br />
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.<br />
</div><br />
You're seeing this because the code is looking for a log4j config file. Copy log4j.properties to the same directory you started the Java server so this error message won't appear.<br />
<br />
After this, your host is now accepting JDBC connections on port 9001.<br />
<br />
8. Here's some sample code that should work:<br />
<br />
<pre class="brush: ruby" name="code">#!/usr/bin/perl
use strict;
use DBI;
my $user = "user";
my $pass = "password";
my $host = "10.10.10.1";
my $port = 9001;
my $url = "jdbc:vjdbc:rmi://$host:2000/VJdbc,eqe"; # Get this URL from JDBC data src
my %properties = ('user' => $user,
'password' => $pass,
'host.name' => $host,
'host.port' => $port);
my $dsn = "dbi:JDBC:hostname=localhost;port=$port;url=$url";
my $dbh = DBI->connect($dsn, undef, undef,
{ PrintError => 0, RaiseError => 1, jdbc_properties => \%properties })
or die "Failed to connect: ($DBI::err) $DBI::errstr\n";
my $sql = qq/select * from table/;
my $sth = $dbh->prepare($sql);
$sth->execute();
my @row;
while (@row = $sth->fetchrow_array) {
print join(", ", @row), "\n";
}
</pre><br />
One last point: your URL and JDBC properties hash are configured with the IP address of the actual JDBC data source. But your data source is configured to talk with localhost. So you establish a JDBC connection to the Java server on localhost on port 9001, and the Java server in turn establishes a connection to the actual data source using the URL.Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-61715256003660381892010-10-04T20:42:00.000-04:002010-12-29T13:04:16.316-05:00Connecting to SQL Server from OS X perlI've been spending my coding time in the offhours working on Perl instead of Ruby. My coding time in general has been very limited, which is part of the reason for the length of time between updates. :)<br />
<br />
My latest project is to pull data out of a Microsoft SQL Server database for analysis. I'm using perl for various reasons: I need a crossplatform environment, and I need certain libraries that only work on perl. Some of the target users for my code run on Windows.<br />
<br />
I know that Ruby runs on Windows but it's not the platform of choice for Ruby developers. The vast majority seem to develop either on OS X or Linux. So Ruby on Windows isn't at the maturity that <a href="http://www.activestate.com/activeperl">ActiveState perl</a> is on Windows.<br />
<br />
In fact, I don't even run native perl anymore on my MacBook Pro. I've switched over to ActiveState perl because I don't need to compile anything every time I want to install new CPAN libraries. And because it's ActiveState, I'm that much more confident it will work on other platforms.<br />
<br />
The bottom line is that perl makes the most sense for what I'm trying to do. I vastly prefer ruby to perl but I don't mind working in perl when I have to.<br />
<br />
So how to connect to SQL Server from perl? My first thought was that I could use ODBC. My research quickly took me to <a href="http://search.cpan.org/%7Emjevans/DBD-ODBC-1.25/ODBC.pm">DBD::ODBC</a> in CPAN. After spending some time Googling for other examples and trying to get it working, I wasn't getting anywhere.<br />
<br />
It took me some more research until I realized that DBD::ODBC is only one piece of the whole picture. I also need an ODBC <a href="http://search.cpan.org/%7Emjevans/DBD-ODBC-1.24/FAQ#What_is_the_ODBC_driver_manager?">driver manager</a> and an ODBC driver.<br />
<br />
The two main ODBC driver managers for Unix/Linux are <a href="http://www.blogger.com/www.unixodbc.org">unixODBC</a> and <a href="http://www.iodbc.org/">iODBC</a>. Fortunately, iODBC is already included in OS X.<br />
<br />
For the ODBC driver itself, I wound up using <a href="http://www.freetds.org/">FreeTDS</a>. Because FreeTDS is only available as source code, I had to use <a href="http://www.macports.org/">MacPorts</a> to download and compile the code. MacPorts installs everything in /opt/local by default.<br />
<br />
So here was the process:<br />
<br />
1. Install FreeTDS. If you are using ActiveState perl as I am, you must force MacPorts to build FreeTDS as 32-bit because ActiveState is 32-bit only. If you go with the default of compiling FreeTDS as 64-bit (or x86_64) then you will get this error when you call the FreeTDS library code from ActiveState perl:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">[iODBC][Driver Manager]dlopen(/opt/local/lib/libtdsodbc.so, 6): no suitable image found. Did find:<br />
/opt/local/lib/libtdsodbc.so: mach-o, but wrong architecture (SQL-00000) at test.pl line 26<br />
</div><br />
So to fix this error, edit the MacPorts configuration file:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">sudo vi /opt/local/etc/macports/macports.conf<br />
</div><br />
Uncomment this line:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">set build_arch i386<br />
</div><br />
Once you build and install FreeTDS, ensure that you can use it to talk with the database. I used tsql:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">TDSVER=8.0 tsql -H 10.10.10.1 -p 1433 -U 'DOMAIN\user' -P 'password'<br />
</div><br />
2. After you do this, test your ODBC driver manager and driver. I used iODBC because it comes with OS X. It comes with a utility called iodbctest. I used similar parameters to tsql when testing:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">iodbctest "Driver=/opt/local/lib/libtdsodbc.so;Server=10.10.10.1;Port=1433;TDS_Version=8.0;uid=DOMAIN\user;pwd=password;Database=Database"<br />
</div><br />
Note a few things:<br />
<ul><li>Other sites tell you to make entries in odbc.ini, odbcinst.ini, or freetds.conf. If you set all parameters on the command line then you don't need to tweak these other config files.</li>
<li>The Driver parameter is set to the full path for libtdsodbc.so. This is the actual ODBC driver. The name or path may differ, depending on your OS and ODBC driver software.</li>
<li>I haven't created an ODBC data source on my Windows SQL Server host so far. Some sites say you need to do this but I found it worked without this.<br />
</ul>3. Use the perl code with DBD::ODBC to call the database. Here's some sample code. Note how I first populate the data source ($dsn) with the parameters I used with iodbctest, and then I pass this to the DBI->connect method. <pre class="brush: ruby" name="code">#!/usr/bin/perl
use strict;
use DBI;
my $user = 'user';
my $pass = 'password';
my $driver = "/opt/local/lib/libtdsodbc.so";
my $db_server = "10.10.10.1";
my $db_name = 'Database';
my $port = 1433;
my $tds_version = "8.0";
my $dsn = join "", ("dbi:ODBC:",
"Driver=$driver;",
"Server=$db_server;",
"Port=$port;",
"UID=$user;",
"PWD=$pass;",
"TDS_Version=$tds_version;",
"Database=$db_name",
);
my $db_options = {PrintError => 1, RaiseError => 1, AutoCommit => 0, };
my $dbh = DBI->connect($dsn, $db_options);
$dbh->{LongReadLen} = 32768;
my $sql = qq/select * from table/;
my $sth = $dbh->prepare($sql);
$sth->execute();
my @row;
while (@row = $sth->fetchrow_array) {
print join(", ", @row), "\n";
}
</pre>
Note on 12/29/10: I added this after the DBI->connect:
$dbh->{LongReadLen} = 32768;
I was getting "Data truncated" errors when accessing ntext fields. This <a href=http://www.easysoft.com/support/kb/kb00090.html>link </a> was a real help and helped me solve the problem.Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com8tag:blogger.com,1999:blog-6554816174365909737.post-29701139808367685172010-10-04T15:51:00.000-04:002010-10-04T15:51:27.782-04:00Thoughts on the iPadThere are many, many posts on the iPad already. I was quite surprised about the intensity of the debate of its merits, or lack thereof.<br />
<p>I've had my iPad for about 5 months now, and I love it. I preordered it and got it on April 30. Since then it has subtly become part of my daily routine.<br />
<p>Here is a list of my uses for it, ordered from most frequent to less frequent:<br />
<ol><li>Reading books<br />
<ul><li>iBooks<br />
<li>GoodReader<br />
<li>Bible<br />
</ul><li>E-mail<br />
<li>Looking at RSS feeds<br />
<li>Watching movies<br />
<li>Games<br />
</ol>I thought it would be interesting to put together a "collage" of interesting blog posts about the iPad. <p>Here is an <a href="http://cheerfulsw.com/2010/ipad-a-staggering-work-of-obvious/">analysis</a> of the fact that the iPad is considered to be nothing new. Its critics seize this as a point of weakness but in reality it is probably a point of strength: <blockquote class="style">Instead of praising the iPad, critics express their disappointment, because they expected more. They expected a genre buster. They expected something they’d never seen before, something beyond their imagination. Something revolutionary. <p>They’re disappointed that the iPad is so… well… unsurprising. “Oh,” they say. “It’s a big iPhone.” <p>It doesn’t matter if they utter that phrase in distaste. That little sand grain of dismissal becomes the core around which will form a pearl of understanding. <p>Steve knows, better maybe than anyone else, that you don’t just slap a product out there and hope it will succeed. You have to prepare people for it, first. <p>And it’s better that people misunderstand a product, at first, than not understand it at all. <p>People won’t buy a product if they can’t understand it immediately. They can’t understand it immediately if their worldview doesn’t already have a readymade place for it. And their worldview won’t have a readymade place for it, if they’ve never seen anything like it before. <p>Steve expertly wields the powerful tool that is the feeling of recognition. </blockquote>I agree with <a href="http://www.patrickweb.com/weblog/archives/2010_04_04.php#ipad_-_part_1">this post here</a> that the iPad will change the model of personal computing over time. <blockquote class="style">the iPad will change the model of personal computing -- not immediately and not for everyone, but for many millions of people the PC will begin to look like a dinosaur. One of my reasons for such a bullish view is the number of skeptics coming forward to say that the iPad is not what it is cracked up to be. Skeptics have been a reliable predictor of the next big thing -- the Internet is too insecure to allow for banking and insurance. WiFi is too expensive and slow and will fizzle. Blogging was to peak out some years ago. Social networking is a fad. The iPad is just a big iPhone. </blockquote>
The desktop was revolutionary in its time, with its "small" form factor and personal monitor. But there is no reason why we should be stuck with this metaphor forever. Even the laptop is just a variation on the desktop theme. <p><br>
<a href="http://db.tidbits.com/article/11152">This post</a> helps explain why the iPad has been so successful lately. You don't think you want it, and then you try it at the Apple Store or a friend's house and then you realize how much you want it. <blockquote class="style">there is a certain magic to using the iPad that's nearly impossible to convey in words - you have to touch it to believe it. the iPad becomes the app you're using. That's part of the magic. The hardware is so understated - it's just a screen, really - and because you manipulate objects and interface elements so smoothly and directly on the screen, the fact that you're using an iPad falls away. You're using the app, whatever it may be, and while you're doing so, the iPad is that app. </blockquote><br>
Some people wonder, will the iPad replace my laptop? I don't think it will. The laptop will always have a place for heavy-duty content creation, coding, etc. But for everyday use and even for travel, I have seen myself use my iPad much more. I'll post in the near future about how I've started taking my iPad on business trips instead of my laptop.
<p>I look forward to seeing what type of <a href="http://www.macworld.com/article/150330/2010/04/ipadreview.html">brand new applications</a> the iPad will enable in the next few years.
<blockquote class="style">Sure, if the interfaces of iPad apps were just scaled-up versions of iPhone apps (like what you get if you run iPhone-only apps on the iPad), the iPad would be the technological equivalent of one of those oversized novelty checks presented to lottery winners. But what the additional pixels really allow is entirely new, richer, and more complex interactions. Beyond the more sophisticated user-interface possibilities, the iPad’s large screen opens the door for new gestures that simply wouldn’t work on a pocketable device. So can the iPad truly replace a laptop? It all depends on what you use your laptop for. The iPad isn’t going to replace a MacBook Pro anytime soon. But let’s face it: there are plenty of tasks that we currently use laptops for (checking e-mail and Twitter, surfing the Web, looking up some actor on IMDB) that don’t really tap the power of a laptop. These are the tasks the iPad is perfectly suited for. </blockquote><br>
The iPad has gotten a lot of criticism for being such a controlled environment. But game consoles are similar in <a href="http://www.dailykos.com/storyonly/2010/4/11/856114/-My-iPad-as-a-tool">how tightly controlled</a> they are. And look how they have transformed the gaming market. I'm the last person to say that PC gaming will die any time soon, but it is only a fraction of the overall gaming market.
<blockquote class="style">Able to control its hardware and software, Microsoft avoided the instability created by the endless hardware/software configurations found on PCs. Not only are users spared the pain of endless crashes, but they don't have to worry about hardware requirements when purchasing games. Rather than fret about whether one has enough Video RAM or processor speed, it's literally plug and play. Of course, that kind of stability has a price. Microsoft requires Xbox developers to register themselves, and all games must be approved by the company before they can be sold to the public. Such rigidity limits the freedom of developers to write for the platform as they see fit, but it allows Microsoft to ensure that end users get the kind of enjoyable experience that keeps them buying Xbox games. Same goes for Sony and the PlayStation, and I'm sure for Nintendo and its Wii as well. In the end, those closed gaming systems have been so effective, that they effectively killed PC gaming. </blockquote><br>
For all of the iPad's advantages, however, there are definitely some challenges. I'm very glad that Apple has recently lifted restrictions on running interpreted code on iPhones and iPads. But there are definitely <a href="http://al3x.net/2010/04/05/ipad-openness-moderates.html">some more suggestions</a> that Apple should seriously consider.
<blockquote class="style">Human-computer interaction has found a sweet spot on the iPad. It’s all the power of desktop computing, plus the valuable constraints of mobile devices, minus the limitations of both. It just makes sense. Use one for a couple hours and your desktop or laptop will seem clumsy, arbitrary, and bewildering. It is, simply, how (most) computing should be. The iPad is a beautiful, important, transformative device released under a confusing regime of questionable ethics. <ol><li>Apple should not charge to put applications you’ve written onto your personal iPad (or iPhone, for that matter).<br />
<li>Apple should lift restrictions on running interpreted code on its mobile devices. Let people run Basic, Python, and Ruby interpreters on iPad and iPhone<br />
<li>Apple should remove the concept of private APIs from its developer offerings. Give developers the same tools that Apple’s own programmers get to use.<br />
</ol></blockquote><br>
I'll end with a <a href="http://roomfordebate.blogs.nytimes.com/2010/04/06/the-ipad-in-the-eyes-of-the-digerati/">final quote</a> from one of my favorite technology analysts, Tim O'Reilly. I've been following him for years and he has generally been dead-on with his insights and predictions.
<blockquote class="style">But the iPad signals more than the end of the PC era. It signals that the App Store, the first real rival to the Web as today’s dominant consumer application platform, isn’t going to be limited to smartphones. It signals that App Store-based e-commerce may replace advertising as the favored model of startup entrepreneurs. It signals that cheap sensors are ushering in an era of user interface innovation. Apple’s Achilles’s heel is that it seems to have come too late to an understanding of the key drivers of lock-in in the Internet era: not hardware, not software, but massive data services that literally get better the more people use them. Media and application syncing across iPhone and iPad is poorly thought out. MobileMe, which should be Apple’s gateway drug for lock-in to Apple services, is instead sold as an add-on to a small fraction of Apple’s customer base. If Apple wants to win, they need to understand the power of network effects in Internet services. They need to sacrifice revenue for reach, taking the opportunity of their early lead to tie users ever more closely to Apple services. </blockquote>Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-80617648329774931592010-07-01T23:34:00.000-04:002010-07-01T23:34:35.946-04:00Converting string to unique integer in perlI'm writing some perl code for a project I'm working on. One of my needs is to convert a string into a 32-bit integer. The catch is that the conversion must be deterministic (i.e. a hash function).<br />
<br />
I looked at using CRC32 but the resulting integer wasn't always the same each time. I looked at using MD5, but MD5 produces a very long hex string that will overflow a 32-bit integer.<br />
<br />
I decided to do something a bit weird; because a 32 bit integer has up to 4,294,967,295 values (signed), I could take the first 8 characters in the MD5 value and then convert any of the letters to a corresponding number.<br />
<br />
So the function looks like this in perl:<br />
<br />
<pre class="brush: ruby" name="code">sub convert_string
# Function to convert string into a unique 32-bit integer
{
my ($str) = @_;
my $md5str = md5_hex($str);
my $md5strsub = substr $md5str, 0, 8;
$md5strsub =~ tr/a-f/1-6/;
return $md5strsub;
}
</pre>Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-36030974612348590552010-06-13T22:18:00.000-04:002010-06-14T12:55:22.834-04:00Relevance of "Social Studies" as a college major to product managementWhenever people ask me what I majored in at college, I always hesitate a bit before answering. The real answer is <a href="http://socialstudies.fas.harvard.edu/icb/icb.do">Social Studies</a>, though I typically don't give this answer anymore. People would either give me a puzzled look or a snide comment, such as "Oh, I took that in 5th grade." So I started answering that I majored in sociology. Not many people really know what sociology is either, but at least it doesn't sound like a 5th grade subject.<br />
<br />
Here's a good <a href="http://webdocs.registrar.fas.harvard.edu/ugrad_handbook/current/chapter3/social_studies.html">explanation</a> of what the Social Studies program is about:<br />
<blockquote class="style">Social Studies is a unique program of study at Harvard College. ...It reflects the belief that the study of the social world requires an integration of the disciplines of history and political science, sociology and economics, anthropology and philosophy. Concerned with the fragmentation caused by increasing disciplinary specialization, the faculty and students of Social Studies seek an integrated approach to the study of social phenomena that synthesizes the findings as well as the methods of various modes of social inquiry.</blockquote>So it's basically a multidisciplinary version of sociology, but much more than that.<br />
<p>I was asked by the university a few months ago to provide some reflections on how majoring in Social Studies has affected my career path as a business person. My contributions were combined with those of people in other fields such as law, medicine, entertainment, journalism, etc.<br />
<p>You can view the completed guide here, titled <a href="http://isites.harvard.edu/fs/docs/icb.topic735046.files/Social%20Studies%20Careers.pdf">Life After Social Studies: Careers</a><br />
<p>I enjoyed writing down my thoughts and wanted to put them here. The questions are in bold and my answers follow.<br />
<p><b>In what ways, if any, did your experience in Social Studies affect your professional choices?</b><br />
<p>I was always interested in entering the technology industry after graduation but I wanted to get a solid liberal arts education while I was in college. I had heard that this type of education develops skills in critical thinking, writing, and information gathering. It can be difficult to build these skills in a work environment, so I wanted to do this as an undergraduate.<br />
<p>Social Studies has the reputation as the "quintessential" liberal arts education, so I was attracted to this concentration. I knew that majoring in liberal arts would make it more difficult to enter the technology industry because new college hires in technology typically have Computer Science or Engineering backgrounds. I believed, however, that I could develop my technology background on my own, either via self-learning or various internships.<br />
<p><b>Did Social Studies provide you with particular skills or experiences that prepared you for your profession? If so, which ones?</b><br />
<p>The most valuable skills I learned from Social Studies were:<br />
<ol><li>Being able to process large amounts of information fairly quickly and glean the trends and patterns<br />
<li>Writing clearly and quickly.<br />
</ol>I do a significant amount of research regarding new technologies or competitive analysis. I need to sift through large amounts of articles or technical manuals to learn what I need to know. Also, I'm constantly writing different types of documents such as product requirement documents, internal communications, marketing descriptions, etc. All those papers I had to write in college, especially the senior thesis, taught me how to quickly get the material out of my head and on to a piece of paper (or computer). <p><b>What career advice do you have for current Social Studies students?</b> <p>Don't limit your career horizons to what is offered at Harvard. Your skillset is broadly applicable to many different industries. Follow what you are most interested in, as opposed to what is being promoted most visibly on campus. The recruiting process at Harvard is somewhat overrepresented by banking and consulting firms. Harvard students naturally think that these are the only professional opportunities available to them outside of grad school, or that these are somehow the best opportunities. It is true that banking and consulting firms do an impressive job of recruiting at Harvard. But there is a world of opportunities outside of these industries.
<p>Don't be afraid to offer your services for free in summer internships, if you can afford this. As a college student without any work experience, one of your biggest competitive advantages will be what you can get on your resume between school years. Many companies may be willing to bring you on as an unpaid intern where they would otherwise not be interested if they had to pay you. As one example, I was able to break into the technology industry because I worked for a company for free during the summer before my senior year.
<p>One final tip: don't overthink the career process. Your first job is a first step in one direction, but it is difficult to know which direction to go in without actually being in the workforce. Be prepared to make many course corrections over time, and this is a standard part of developing a career.Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-75736677772248021062010-04-22T00:10:00.000-04:002010-04-23T23:35:56.486-04:00Measuring memory per Unix processWow, it's been a long time since I've written anything here. Since my last blog post, I've been taking two classes (in two very different topics), along with the other daily responsibilities of working, raising two children, etc.<br />
<br />
For a work-related requirement, I had to figure out how to measure memory consumed by a specific Unix/Linux process. This is more difficult than it may seem.<br />
<br />
For one thing, apparently the memory statistics given by Linux are <a href="http://mail.gnome.org/archives/gnome-list/1999-September/msg00036.html">meaningless.</a> This is mainly because the ps command (and the VSZ metric specifically) only lists the size of the address space referenced by a process, not the actual memory size itself. <a href="http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html">This</a> page suggests the use of smaps, where /proc/$pid/smaps provides the actual amount of memory used by a process.<br />
<br />
Because the output of smaps is pretty lengthy, <a href="http://wingolog.org/archives/2007/11/27/reducing-the-footprint-of-python-applications">Someone</a> wrote a python script called mem_usage.py to make the output more understandable.<br />
<br />
The main issue is that smaps only exists in Linux, and I had a requirement to measure memory usage in OS X too. OS X doesn't even support the /proc concept so I couldn't use smaps.<br />
<br />
In the end, I wound up going back to ps and relying on the RSS (resident set size) metric. It's displayed on multiple operating systems and indicates how much RAM is being used for the text and data segments for a specific process in kilobytes.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">11:52:44[~/ruby:162]$ ps u</span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND</span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">rpark 2668 0.6 1.4 1652720 44816 s000 S 12:40AM 5:01.71 ./eclipse</span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">rpark 4598 0.0 0.1 2435088 1800 s001 S+ 11:01PM 0:00.09 ssh rpark@1</span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">rpark 2878 0.0 1.2 2769524 36688 s000 S 1:17AM 0:25.44 /System/Lib</span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">rpark 799 0.0 0.0 2435468 792 s001 S Tue12PM 0:00.18 -bash</span></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="font-size: small;">rpark 535 0.0 0.0 2435468 756 s000 S Tue11AM 0:00.30 -bash</span></span><br />
<br />
<a href="http://stackoverflow.com/questions/131303/linux-how-to-measure-actual-memory-usage-of-an-application-or-process">Here</a> are some <a href="http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html">more</a> discussions on this topic.Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-32821329916070836752009-11-13T12:38:00.000-05:002009-11-13T13:28:42.470-05:00Merits of Customer DevelopmentI wouldn't be a card-carrying product manager without some thoughts on product management.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://weblogs.cltv.com/news/local/chicago/smiley%20face-thumb.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://weblogs.cltv.com/news/local/chicago/smiley%20face-thumb.jpg" /></a><br />
</div><br />
My latest product management interest has been following a trend known as Customer Development, advocated by <a href="http://steveblank.com/about/">Steve Blank</a>. It's a business model for developing new products, mostly applicable to startups but I think it also applies to more established companies.<br />
<br />
The main message is actually fairly intuitive: you develop a successful product by continually iterating it in a tight feedback loop between developing the product, getting customer input, and then making changes. Rather than spend a lot of time upfront in creating the product, you develop a <a href="http://venturehacks.com/articles/minimum-viable-product">minimum viable product (MVP)</a>: the product with just the necessary features to get money and feedback from early adopters. Then you let your early adopter customers tell you what works well and what needs to be changed.<br />
<br />
This model sounds intuitive but by far the most prevalent development model for Silicon Valley startups looks something like this:<br />
<br />
1. Get excited about an idea. Start doing some research regarding markets, customers, pricing, etc.<br />
2. Build the product, along with accompanying sales tools (demo, PowerPoint slides, data sheets, etc.). Start building a sales force to sell the product.<br />
3. Work with a small group of alpha/beta customers. Enlist a PR agency start building "buzz."<br />
4. Officially release the product in a public launch event, hopefully getting lots of attention from a site such as Digg or Techcrunch. Go full steam ahead in selling and marketing the product.<br />
<br />
Steve Blank calls the model I just described as the Product Development model, and he labels it <a href="http://steveblank.com/2009/08/27/the-leading-cause-of-startup-death-the-product-development-diagram/">"the leading cause of startup death"</a>. You can read more about why on his blog, starting with <a href="http://steveblank.com/2009/08/27/the-leading-cause-of-startup-death-the-product-development-diagram/">this link</a>.<br />
<br />
I've started reading Steve's book, <a href="http://www.cafepress.com/kandsranch">Four Steps to the Epiphany.</a> I first heard about the book on <a href="http://pmarca-archive.posterous.com/book-of-the-week-best-book-for-tech-entrepren">Marc Andreessen's blog</a>. I was intrigued by his recommendation because Marc has some really great thoughts on the idea of <a href="http://pmarca-archive.posterous.com/the-pmarca-guide-to-startups-part-4-the-only">product-market fit</a> - the idea that what matters most in determining the success of a product is how much it fits what the market needs. It turns out that Marc was borrowing concepts from Steve Blank's book!<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://www.cafepress.com/kandsranch" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://images5.cafepress.com/product/58024175v1_240x240_Size3Front.jpg" /></a><br />
</div><br />
The book is laid out more like a manual but from what I can tell, it has great content.<br />
<br />
In fact, I don't know why we don't hear more about Steve Blank and Customer Development. The model makes so much sense because it basically says that entrepreneurs may have vision but they aren't fortune tellers. They can't predict exactly what people need so there is a constant need to go back and iterate.<br />
<br />
Maybe this model isn't so popular because it flies in the face of conventional wisdom? Imagine if all startups started to follow this model of starting with a <a href="http://venturehacks.com/articles/minimum-viable-product">minimum viable product</a> and holding off on enlisting the professional sales force, marketing team, PR agency, etc. upfront. This would certainly change the economics of the startup industry in regions such as Silicon Valley.<br />
<br />
Here's another fascinating article inline with the idea of Customer Development. It seems almost strange that a Web site could make 50 changes in their production system every day, but if you take the time to read through the article, it makes a lot of sense.<br />
<br />
<a href="http://timothyfitz.wordpress.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/">http://timothyfitz.wordpress.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/</a>Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-83422852437906255322009-11-12T10:31:00.000-05:002009-11-12T14:15:50.432-05:00Displaying Ruby code on a blogI spent a bit of time trying to figure out the best way to display Ruby code on this blog. Initially I just converted the code to Courier font but it looked ugly and was hard to work with.<br />
<br />
I eventually found this link on Stack Overflow:<br />
<br />
<a href="http://stackoverflow.com/questions/1644201/how-can-i-display-code-better-on-my-blogger-blog">http://stackoverflow.com/questions/1644201/how-can-i-display-code-better-on-my-blogger-blog</a><br />
<br />
There were a few other methods I found, such as <a href="http://blog.wolfman.com/articles/2006/05/26/howto-format-ruby-code-for-blogs">this strategy</a> to convert the code via a Ruby script to HTML and then put it in the clipboard. But I liked the Stack Overflow method the best because all I need to do is add these HTML tags to the code:<br />
<br />
<pre class="brush: ruby" name="code"><br />
(Code)<br />
<pre><br />
<br />
Here's an example of what it looks like:<br />
<br />
<pre class="brush: ruby" name="code">Class Foo
def bar
end
end
</pre><br />
And here is how to widen the main text column so the code doesn't constantly get wrapped:<br />
<br />
<a href="http://johndeeremom.blogspot.com/2008/07/how-to-widen-your-columns-on-blogger.html">http://johndeeremom.blogspot.com/2008/07/how-to-widen-your-columns-on-blogger.html</a>Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-52541070218315278622009-11-11T11:36:00.000-05:002010-04-21T22:50:42.276-04:00Build simple PDF search engine in Ruby (Part 1)I decided to build a simple Ruby search engine to search through PDFs.<br />
<br />
The main application was that I wanted a quick way to search through songsheets on my church's Web site. I didn't want to repeatedly look through different PDFs to find the song I was interested in.<br />
<br />
I was mostly inspired by <a href="http://blog.saush.com/2009/03/17/write-an-internet-search-engine-with-200-lines-of-ruby-code/">this example</a> of someone who had written a search engine in 200 lines of Ruby. I knew my program would be much easier because it didn't need to support any crawling; just indexing and querying.<br />
<br />
The first challenge was to find a Ruby library that would parse PDFs. I ultimately settled on <a href="http://github.com/eterps/pdf-struct">this</a> because it was easy to work with. It's basically just a Ruby wrapper around pdftohtml that provides high level access to the text objects of a PDF. I don't care about layout, graphics, etc. so this was sufficient.<br />
<br />
The PDF code mostly works without problems but it assumes that the directory for pdftohtml exists in $PATH. I used MacPorts to compile pdftohtml so it was stored in /opt/local/bin, and TextMate didn't recognize /opt/local/bin in my $PATH. I did some research and discovered <a href="http://manual.macromates.com/en/shell_commands">this page</a> that says I need to create a file called ~/.MacOSX/environment.plist and explicitly set the PATH variable:<br />
<br />
<pre class="brush: ruby" name="code">{
PATH = "/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin";
}
</pre><br />
The actual indexing code is straightforward. It's mostly based on the saush engine article. Rather than rehash the site, the index is based on an <a href="http://en.wikipedia.org/wiki/Inverted_index">inverted index</a>. The search engine saves the inverted index in a SQLite database using the DataMapper library.<br />
<br />
There are three main "tables": Song, Word, and Location. Song and Word have a many-to-many relationship, where a song has multiple words and a word is used in multiple songs. Location is the mapping table between Song and Word.<br />
<br />
Here is the indexing library. Note that it uses DataMapper so it relies on the dm-core and dm-timestamps libraries, as well as stemmer and pdf-struct (the PDF library mentioned earlier). The saush search engine uses dm-more but I couldn't get this to be properly included. But dm-timestamps was all that was needed out of dm-more.<br />
<br />
Here is the code for index.rb:<br />
<br />
<pre class="brush: ruby" name="code">require 'rubygems'
require 'dm-core'
require 'dm-timestamps'
require 'dm-aggregates'
require 'stemmer'
require 'pdf-struct'
DBLOC = 'songdb.sqlite3'
DataMapper.setup(:default, 'sqlite3:///' + DBLOC)
class String
def words
words = self.gsub(/[^0-9A-Za-z_\s]/,"").split # self is the string; no need for parms
# Get rid of all non-word and non-space characters and split on spaces
d = []
words.each { |word| d << word.downcase.stem unless word =~ /^[A-G]+[bgm]?$/ } # Ignore guitar chords
return d
end
end
class Song
include DataMapper::Resource
property :id, Serial
property :title, String, :length => 255
has n, :locations
has n, :words, :through => :locations
property :created_at, DateTime
property :updated_at, DateTime
def self.find(title)
song = first(:title => title)
song = new(:title => title) if song.nil?
return song
end
def refresh
update( {:updated_at => DateTime.parse(Time.now.to_s)})
end
end
class Word
include DataMapper::Resource
property :id, Serial
property :stem, String
has n, :locations
has n, :songs, :through => :locations
def self.find(word)
wrd = first(:stem => word)
wrd = new(:stem => word) if wrd.nil?
return wrd
end
end
class Location
include DataMapper::Resource
property :id, Serial
property :position, Integer
belongs_to :word
belongs_to :song
end
DataMapper.auto_migrate! if ARGV[0] == 'reset' # This issues the necessary Create statements and wipes out existing database
</pre><br />
The actual indexing code goes through each PDF. It extracts the words from the song (except the guitar chords) and creates a space-delimited string of words. Then it goes through the string, creating the Word or Song objects if necessary and creating the many-to-many relationship between Word and Song.<br />
<br />
Code for pdfindex.rb:<br />
<br />
<pre class="brush: ruby" name="code">#!/usr/bin/ruby
require 'rubygems'
require 'fileutils'
require 'logger'
require 'index'
SONGDIR = '/Users/rpark/ruby/pdfsearch/'
LOGFILE = 'songsearch.log'
LASTRUN = 'lastrun'
class SongSearch
def process(file) # returns string of all stemmed words in song
array = []
document = PDF::Extractor.open(file)
document.elements.each do |element|
array << element.content
end
return array.join(" ").words # .join creates a string separated by delimiter
rescue => e
#puts "Exception in parsing #{e}"
@log.debug "Exception in parsing #{e}"
nil
end
def index(words, filename)
if words.nil?
#puts "ERROR parsing #{filename}"
@log.debug "ERROR parsing #{filename}"
return
end
print "Indexing #{filename}: "
logmsg = "Indexing #{filename}: "
song = Song.find(filename)
unless song.new?
print "Overwriting... "
logmsg += "Overwriting... "
song.refresh
song.locations.destroy!
end
words.each_with_index { |word, index|
loc = Location.new(:position => index)
loc.word, loc.song = Word.find(word), song
loc.save
}
puts "#{words.size.to_i} words"
@log.debug logmsg + "#{words.size.to_i} words"
end
def cycle
lastrun = File.mtime(LASTRUN)
@log = Logger.new(LOGFILE, 'monthly')
Dir.glob(SONGDIR + "*.pdf") {
|file|
index(process(file), file) if File.mtime(file) > lastrun # Only process newer songs
}
FileUtils.touch LASTRUN
end
end
search = SongSearch.new
search.cycle
</pre><br />
The digger code actually searches through the song database and searches for songs. A song is searched for by passing a string to Digger.search(<string>). It returns a list of songs that the string can be found in, along with a score.<br />
<br />
Code for digger.rb:<br />
<br />
<pre class="brush: ruby" name="code">#!/usr/bin/ruby
require 'index'
class Digger
SEARCH_LIMIT = 19
def search(for_text)
@search_params = for_text.words
wrds = []
@search_params.each { |param| wrds << "stem = '#{param}'" }
word_sql = "select * from words where #{wrds.join(" or ")}"
@search_words = repository(:default).adapter.query(word_sql)
tables, joins, ids = [], [], []
@search_words.each_with_index { |w, index|
tables << "locations loc#{index}"
joins << "loc#{index}.song_id = loc#{index+1}.song_id"
ids << "loc#{index}.word_id = #{w.id}"
}
joins.pop
@common_select = "from #{tables.join(', ')} where #{(joins + ids).join(' and ')} group by loc0.song_id"
rank[0..SEARCH_LIMIT]
end
def rank
merge_rankings(frequency_ranking, location_ranking, distance_ranking)
end
def merge_rankings(*rankings)
r = {}
rankings.each { |ranking| r.merge!(ranking) { |key, oldval, newval| oldval + newval} }
r.sort {|a,b| b[1] <=> a[1]}
end
def frequency_ranking
freq_sql= "select loc0.song_id, count(loc0.song_id) as count #{@common_select} order by count desc"
list = repository(:default).adapter.query(freq_sql)
rank = {}
list.size.times { |i| rank[list[i].song_id] = list[i].count.to_f/list[0].count.to_f }
#puts freq_sql
#puts list
#puts rank.inspect
return rank
end
def location_ranking
total = []
@search_words.each_with_index { |w, index| total << "loc#{index}.position + 1" }
loc_sql = "select loc0.song_id, (#{total.join(' + ')}) as total #{@common_select} order by total asc"
list = repository(:default).adapter.query(loc_sql)
rank = {}
list.size.times { |i| rank[list[i].song_id] = list[0].total.to_f/list[i].total.to_f }
#puts loc_sql
#puts list
#puts rank.inspect
return rank
end
def distance_ranking
return {} if @search_words.size == 1
dist, total = [], []
@search_words.each_with_index { |w, index| total << "loc#{index}.position" }
total.size.times { |index| dist << "abs(#{total[index]} - #{total[index + 1]})" unless index == total.size - 1 }
dist_sql = "select loc0.song_id, (#{dist.join(' + ')}) as dist #{@common_select} order by dist asc"
list = repository(:default).adapter.query(dist_sql)
rank = Hash.new
list.size.times { |i| rank[list[i].song_id] = list[0].dist.to_f/list[i].dist.to_f }
#puts dist_sql
#puts list
#puts rank.inspect
return rank
end
end
</pre>
<br />
Note: the biggest disadvantage with this search method is that it doesn't show the search string in its context in the song. Rather than continue with this approach, my thinking is to use a search engine such as Solr to do the search, so I can show the search string within the song.Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0tag:blogger.com,1999:blog-6554816174365909737.post-40837611513081412009-10-21T12:10:00.001-04:002017-04-25T00:09:06.949-04:00Purpose of this blogHi, I wanted to create this blog to document some of the coding work I'd like to do, mostly in Ruby and iPhone development. This is mostly for me so I don't forget what I work on, but I hope it's helpful to anyone out there.<br />
<br />
I'm still primarily interested in Ruby on Rails but Sinatra is looking very interesting as a simple Ruby framework for building applications without a database.<br />
<br />
Stay tuned for more stuff!Richard Parkhttp://www.blogger.com/profile/11419620257213286880noreply@blogger.com0