Gathering research and documents related to the Cover Oregon website development failure.
- Where's the source code? I assume someone has already tried to FOIA it and I just haven't found it. If this was a failed infrastructure project citizens would be able to see the partial completion of the bridge or road to at least know how far along the project was. I have more research to do, but I have no sense for how close they got to actually launching the site.
- What's been released as part of the ongoing lawsuits? There's 6 different lawsuits related to the project. How much was publicly released as part of those trials?
- The system was unable to update changes after initial enrollment. This design failure was called a "fatal" flaw that doomed the project from the beginning.
- Oracle initially promised that only 5% of their software would need to be customized for the site. It was later reported to be closer to 40%.
Note: I'm using Markdown headers to create anchor links within the document. For example, #### article1
creates an anchor which can be linked to with a Markdown link like [example link](#anchor1)
.
To keep track of all the interrelated people and concepts I'm going to attempt to store information in linked Markdown files using Obsidian.
Complaint from the state accusing Oracle of: "Fraud; Oregon False Claims Act; Breach of Contract; Oregon Racketeer Influenced and Corrupt Organtzations Act". Filed on 8/22/14.
[Updated Jan 09, 2019; Posted Apr 08, 2016] https://www.oregonlive.com/business/2016/04/oregon_vs_oracle_war_gets_pers.html
- "Oracle claims the state tipped off a reporter about how to remove redactions in state documents hiding snippets of damaging Oracle internal emails."
- Where are these unredacted emails?
- "Oracle was the primary contractor"
- Who are the secondary / other contractors?
- "The dispute has led to six lawsuits, including the main case playing out in Marion County Circuit Court, in which the state has accused Oracle of fraud and racketeering and is seeking $6.5 billion in damages."
- What's the outcome from these 6 different lawsuits.
- "Oracle was paid over $240 million to deliver an exchange that never worked"
- That's more than the $200M figure I've heard previously.
- "By March 2014, after years of development and more than $300 million in costs, then Gov. John Kitzhaber pulled the plug on the Cover Oregon technology effort."
- Does that $300M figure include the state's costs?
- "Oracle's narrative is radically different. It claims its team delivered a functional health care exchange by February 2013. But Kitzhaber opted to dump the technology and claim it never worked."
- Um what? It "worked" but they never shipped it? Seems like they'd want to show / leak some proof of that if it's true.
- "Oracle launched a lobbying campaign intended to convince local business, political and media officials that the state was botching the Oracle litigation nearly as badly as it bungled the Cover Oregon effort."
- Was all this lobbying private / behind closed doors? Is there any evidence of this lobbying?
- "[Oracle] has categorized 97 percent of the documents it provided to the state in the discovery process as confidential or for attorney's eyes only."
- Are the remaining 3% at least publicly available?
- "In January, the state asked Marion County Circuit Court Judge Courtland Geyer to force Oracle to declassify some of those half a million pages of documents that "support plaintiff's claims and undermine Oracle's false media campaign that the health insurance exchange 'worked' in February 2014."
Oracle argued the state simply wanted to leak the documents to reporters, who would write stories embarrassing to the company."
- More public information would be great.
- "He castigated his own company for bringing an "army" of programmers to Oregon who were "rapoing [sic] the state f [sic] Oregon on something that will never work well."
Oracle claimed the email was taken out of context."
- What other context would improve this remark?
- "Rather than wait for January and the scheduled start of the trial, he [Geyer] has recently ordered the combatants into mediation."
- Does that mean fewer documents and proceedings will be made public?
[Nick Budnick Portland Tribune Feb. 22, 2016 5:30 p.m.] https://www.opb.org/news/article/documents-oracle-doesnt-want-you-to-read-about-cover-oregon/
- Audio interview with Nick Budnick.
- "State of Oregon does not come out looking good in these"
- "Oracle's work fails the laugh test" ~ Oracle employee
- Oracle blames the State for failing to hire an oversight contractor. Which the State rejects because they claim the company recommended they NOT hire an oversight contractor.
- March 2012 Oracle exec is quoted in an internal email that they tried to keep the State from hiring an oversight contractor.
[Nick Budnick Feb. 22, 2016]
- "Last month [Jan 2016] the Oregon attorney general’s office submitted a 24-page brief in Marion County Circuit Court arguing that Oracle was wrongly classifying its internal documents as confidential."
- Link to brief?
- "Last week, however, the Pamplin Media Group gained access to a version of the brief without the redactions."
- Very interesting.
- "One company assessment in November 2013 found its software development “didn’t pass the ‘laugh test.’”"
- "That a Nov. 22, 2013, internal email appears to contain two typos, one Oracle developer wrote that an “army” of programmers for the company was “rapoing the state f Oregon on something that will never work well.”"
- Wow.
- "concerns about the project prompted Oracle’s founder, billionaire Larry Ellison, to request daily progress reports in a Nov. 16, 2013, email."
- That's crazy if it escalated all the way up to Ellison.
- How big was this project for Oracle? What are their annual revenues?
- In FY13 Oracle had $27.4B in software revenues - which a assume this falls under because their other line items are hardware and services. https://s1.q4cdn.com/289076952/files/doc_financials/quarterly/2014/q2fy14-detailed-financials-2087031.pdf
- "“Oracle has never worked on a project of this size and complexity without a systems integrator in place,” according to a PowerPoint presentation Oracle shared with members of Congress."
- "If the state hired an integrator, it “could displace or reduce our role significantly,” wrote the Oracle executive, according to a version of the state’s brief: “It isn’t clear what we can do about this other than pray that they keep deferring the SI (systems integrator) selection. But since hope is not a strategy, this topic deserves some discussion.”"
- quote from an unnamed Oracle executive.
- "As the company argued in its federal lawsuit, “That decision was akin to an individual with no construction experience undertaking to manage the processes of designing and building a massive multi-use downtown skyscraper without an architect or general contractor.”"
- But Oracle claimed they would fill that role, at least the State claims.
- "The public could get a chance to see the full documents after March 10, when a Marion County Circuit Court judge will consider whether to unseal them."
- Where are these documents?!
- "Deollitte got paid a nice amount as well to come in later and try to fix the IT problems.
Several CO people quit and sued the State, not Oracle.
Why is that?"
- Are there any records showing Deloitte's perspective?
- Did former contractors sue the state directly?
[Jeff Manning, Updated Jan 09, 2019; Posted Apr 12, 2017] https://www.oregonlive.com/politics/2017/04/state_to_pay_13_million_to_off.html
- "Carolyn Lawson contends she was made a scapegoat for the failure of the state and its contractors to construct a working health care exchange website. The former chief information officer for the Oregon Health Authority claimed senior state officials covered up the truth, content to let her take the fall."
- "She quit in December 2013 under threat of getting fired."
- "She's been unable to find work in information technology due to her role in Cover Oregon, her lawyers said."
- What is she doing now?
- Is this the same person? @carolynlawson
- What is she doing now?
- "The two sides settled in September, with Oracle agreeing to pay the state $25 million in cash and another $75 million worth of software and services."
- Wow so is that the final resolution of everything? That's a tiny pay out, compared to the damages they were seeking initially.
- ""Caroline feels vindicated by the results in this case," said Kyle Busse, one of her attorneys. "The amount the state was willing to pay speaks for itself.""
[Sean Gallagher, 8/25/2014, 9:34 AM] https://arstechnica.com/tech-policy/2014/08/oregon-attorney-general-sues-oracle-for-racketeering-activity/
- "The 126-page complaint, filed by Rosenblum in Oregon’s Marion County Circuit Court on August 22, claims that Oracle pushed the state to not hire a systems integrator for the project, giving the company total control over the development of the site and allowing company executives to conceal problems with the software."
- Thankfully they included a link to the actual filing! Interesting that it's hosted by OregonLive.
- "... which allowed the company to conceal problems with software and essentially bill for fixes to the software that should have been covered under the license agreements for the software itself, the attorney general asserted."
- Is that because the underlying system Oracle was re-using were shared software systems created by Oracle for other project? Essensially, billing for fixes to the "commons" part of the software?
[Jeff Manning, Updated Jan 10, 2019; Posted Aug 08, 2014] https://www.oregonlive.com/business/2014/08/oracle_cover_oregon_lawsuit.html
- "Cover Oregon sued Oregon on Friday seeking $23 million."
- ? What? Cover Oregon sued Oregon or Oracle? And why only $23M?
- "In the complaint filed Friday, Oracle says the state's poor management doomed the exchange."
- They included a link to the complaint!
- "In the lawsuit, Oracle points out, First Data, a company hired by the state to analyze what went wrong with the exchange project, "attributed a large part of the problems to Cover Oregon's mismanagement of the project.""
- Who is First Data?
- Their report is linked in notes below.
- Who is First Data?
- "Most Oracle contractors finally left the project this spring, but not before it extracted another $43.9 million from the state. In all, it was paid about $130 million. In the lawsuit, it claims it is owed at least another $23 million."
- So that would be spring of 2014.
- "The company was hired by the Oregon Health Authority in the summer of 2011 as the lead technology contractor on on the exchange."
- Basically a 3 year project before it turned to litigation.
[Colin Miner, Sep 15, 2016 2:13 pm ET] https://patch.com/oregon/portland/oregon-settles-oracle-over-cover-oregon-fiasco
- "After more than one year of fighting, Oregon and Oracle have settled six lawsuits that stemmed from the Cover Oregon health exchange website fiasco."
- So yeah, the settlement wrapped up all the pending litigation.
- "[the State has] spent more than $16 million to four law firms to fight the company."
- Which 4 firms?
- ""Today's settlement agreement ends years of turmoil and taxpayer expense related to a troubled health exchange program I dissolved in March 2015," Governor Brown said on Thursday."
- Important date for the timeline.
- "The settlement includes $25 million cash as well as a six-year Unlimited License Agreement (ULA) for products and services that can be leveraged by the Legislature to significantly modernize state government's IT systems, potentially saving the State hundreds of millions of dollars."
- The ULA just means use of any Oracle product?
- "The agreement also calls for $60 million in free customer service support, a $10 million grant for science, technology, engineering and math (STEM) programs in Oregon schools."
- So some of this money didn't even go back to the government, just loosely back into the state.
- "While the state had won several early rounds against Oracle - include successfully keeping the case from being moved to federal court from state court - there was no guarantee that the state would prevail."
[Nick Budnick, Thursday, August 11, 2016] https://pamplinmedia.com/pt/9-news/318118-194349-cover-oregon-shows-need-for-change-insider-says
- "“I think people should understand why it failed,” Walsh says matter of factly, given how many people worked so hard on the ambitious project, and how much was spent."
- I agree.
- "The reality, according to Walsh? Compared to other large projects, “It should have been easy.”"
- "But he [Walsh] and his Oracle counterparts were repeatedly rebuffed in 2012 when they sought information from other parts of Oracle that they considered crucial to the project’s success, such as how massive quantities of data would fit together under the project’s design, he says."
- "After a while, Walsh pursued a separate data design for his portion of the project to ensure his team was not held up. He credits that decision for his team’s success, as the financial side of Cover Oregon worked fine and was completed on time."
- So there were definitely some parts that worked at some point.
- "In contrast, Walsh still recalls the shock his team felt in July 2013 upon seeing the design for how enrollment would work for the project. A colleague’s muttered reaction: “Well, that’s not going to work.”"
- "Walsh was surprised to hear from Oracle in fall 2012 that the health insurance system its staff envisioned had no capacity to process mid-year changes in a family’s health policy, premiums or tax credits after they enrolled. ... The response? We’ll get to it later."
- So they weren't even aiming to implement everything initially... makes sense if you're shipping an MVP, but this was big ticket.
- "Not until September 2013, the month before the project was supposed to go live, did Walsh and his team learn that Oracle’s design for the health insurance project still did not allow changes to a family’s policy or circumstances. This meant Cover Oregon would not be able to share data effectively with the insurance carriers it worked with — a basic problem that caused ripple effects throughout the project’s workings. ... This, Walsh believes, was the “fatal flaw” of Cover Oregon, and why the exchange had to be scrapped."
- "“The whole thing was going to have to be rewritten ... It was truly unbelievable that it would ever be designed that way,” Pettit told congressional investigators, adding that when he complained to Oracle about its design, the response was, “Well, it wasn’t in the specification.” Pettit’s retort: “Well, it didn’t need to be in the specification. You knew you had to keep track of changes to records, and the system wouldn’t keep track of it.”"
- "For instance, the state spent months and millions of dollars trying to perfect a new type of interface to shift data between two components of the project. Walsh urged the use of a standard Oracle product instead. In the end, when an Oracle analyst confirmed to Cover Oregon managers that Walsh’s idea would work, a top official asked how long it would take to set up the solution."
- What does this mean technically, like what new data process where they trying to create?
- "... management obsessed on the website’s appearance while ignoring the details of how it would work."
- That's not shocking, but sad to hear.
- "Another problem that seems endemic to Oregon IT projects is cronyism, rather than hiring the best-qualified staff, Walsh says."
- That's a big charge.
- "“They hire people they know,” Walsh says, “And then, when things don’t go well, they hire more people they know.”"
- Great quote.
[Chelsea Kopta, Monday, June 23rd 2014] https://katu.com/news/politics/oracle-claims-it-has-proof-cover-oregon-website-worked
- Included video news segment
- Chelsea Kopta says they requested copies of the test case recordings from Oracle. Doesn't say if they ever got them.
- There's an interview with what looks like an Oracle spokesperson but no name or lower third is shown to say who the person is.
- They have recordings from a webinar showing how the system was intended to be used.
- There's images of a presentation Oracle made to US House Commerce and Energy committee.
- "Cover Oregon, unveiled a partial or "soft" launch, 141 days after it was supposed to go live."
- "Cover Oregon also sent an email to agents that described the online portal."
- The link to the email is broken. The file is just named "Dear+Agents.pdf".
- ""Oracle has been ready to release the website for public enrollment since February 2014, and urged Cover Oregon to do so, but Cover Oregon would not permit it," the presentation stated."
- Very interesting claim from Oracle.
- ""Oracle is involved in a dozen exchanges," the presentation continued. "Oracle technology runs mission critical systems globally. Nowhere around the world does Oracle have issues with the customer like we face in Oregon.""
- Whoa! The bottom of the article has over 20 links to reporting by KATU on the project failure. None of the links seem to work, but I'm curious if they're still available through archive.org...
[Nigel Jaquiss, Published September 29, 2018 Updated September 30, 2018] https://www.wweek.com/news/2018/09/29/knute-beuhler-hits-kate-brown-for-cover-oregon-but-that-fiasco-ended-before-she-became-governor/
- "In fact, while Brown served as secretary of state, her audits division commissioned an independent audit of Cover Oregon, which was published in March 2015, the month after Brown succeeded Kitzhaber as governor."
- Link to audit report! https://s3.amazonaws.com/arc-wordpress-client-uploads/wweek/wp-content/uploads/2018/09/29172915/coveroregonaudit.pdf
- Includes information about costs and budgeting. THe meat of it.
- Link to audit report! https://s3.amazonaws.com/arc-wordpress-client-uploads/wweek/wp-content/uploads/2018/09/29172915/coveroregonaudit.pdf
[Kristian Foden-Vencil, March 20, 2014 3:41 p.m. | Updated: March 27, 2014 10:38 a.m.] https://www.opb.org/news/series/vitalsigns/first-data-report-on-cover-oregon-problems/
- Audio from OPB.
- Kristian Foden-Vencil is the one answering questions about the reporting.
- Bruce Goldberg resigning.
- "the company told the state that only about 5 percent of it’s software would need custom configuration. In the end, it was more like 40 percent."
- "Oracle specified that bills were to be paid on time and not tied to the completion of services. First Data said this approach departs from best practices and provided little financial accountability."
- First Data interviewed 67 people, but not many Oracle people were made available.
- "Well the federal Government Accountability Office has its own study into Cover Oregon and other state health care websites. So we’ll wait for that to come out."
- Where is that report?
[Andrew Clevenger, March 6, 2014 at 08:48AM] https://www.bendbulletin.com/home/1853494-151/gao-to-investigate-cover-oregon
- "The GAO confirmed its decision in separate letters Wednesday to Rep. Greg Walden, R-Hood River, and Sens. Jeff Merkley and Ron Wyden, both D-Ore."
- "According to the Centers for Medicaid and Medicare Services, Oregon received more than $300 million in federal grants to create the Cover Oregon exchange. That includes a $226.4 million phase two grant in January 2013 to cover expenses related to testing, training and implementation of the online exchange."
[PR from Governer's office, March 20, 2014] https://www.oregon.gov/newsroom/Pages/NewsDetail.aspx?newsid=579
- "The report, based on 67 stakeholder interviews and the review of more than 3,200 documents"
- That sounds like a very small fraction of the total documents.
- "The Governor called for the independent, third-party review of Cover Oregon in December 2013. First Data began its assessment in January, and the full, un-redacted report was released today on the Department of Administrative Services' website."
- Where's the link people...
- Found through search - https://katubim.s3.amazonaws.com/co_assessment%281%29.pdf
- Exhibit 1 at the end of the report includes a huge list of all the people they interviewed for the report!
- Bob Wort – Cover Oregon User Acceptance Test Manager
- Edward Screven (and Attorney) – Oracle Chief Corporate Architect
- Exhibit 1 at the end of the report includes a huge list of all the people they interviewed for the report!
- Found through search - https://katubim.s3.amazonaws.com/co_assessment%281%29.pdf
- Where's the link people...
- "The Governor has convened a team of information technology experts to advise Cover Oregon on IT options going forward. The team includes the Chief Information Officers of Legacy Health, Pacific Source, Kaiser Permanente, Providence, Moda, Bridgespan/Regence, and Oregon's Chief Information Officer."
- Who are all these people? What did they review and advise?
[Jeff Manning, Updated Jan 10, 2019; Posted Apr 15, 2014] https://www.oregonlive.com/health/2014/04/cover_oregon_new_interim_execu.html
- "Cover Oregon late last week severed one of its contracts with technology consultant Deloitte, the first of what could be a series of cost-cutting moves as the troubled operation enters a new era of fiscal austerity."
- So Deloitte came in at the end to try and clean things up.
- "Clyde Hamstreet, the consultant recently hired to take over executive director duties at Cover Oregon"
- "Deloitte has been working on Oregon's exchange for months. It designed the user-interface on the exchange's website. To date, it's been paid $7.65 million for that work."
- That's a lot less than the price tag for Oracle.
- "Deloitte was tight-lipped about the suspension. "Deloitte fulfilled its contractual responsibilities to Cover Oregon and completed its assessment of the state's health insurance marketplace," said company spokesman Paul Dunker."
- "Cover Oregon signed Hamstreet to a contract that pays his firm a maximum of $100,000. His stint at the organization is likely to last four to six weeks."
- Interesting that its so short.
- Jeff Manning, The Oregonian/OregonLive
- Nick Budnick, Portland Tribune reporter (formerly at OregonLive)
- "broke many of the first stories about the Cover Oregon mess" ~ OregonLive story 1
- Chelsea Kopta, KATU 2 reporter
- Looks like she's now at National Geographic
- John Kitzhaber, former Governer
- He canceled the project in March 2014.
- Kate Brown, current Governer
- Involved in settlement offers and litigation.
- Ellen Rosenblum, State Attorney General
- According to this OregonLive story Rosenblum refused the governers order to settle the case for $25M.
- Kristina Edmunson, Rosenblum's spokesperson
- Brian Shipley, former chief of staff for Kate Brown
- "Once Brown replaced him, the Oracle mess landed in her lap. She dispensed her then-chief of staff Brian Shipley to settle the case." ~ OregonLive story 1
- Kristen Grainger, Brown's spokesperson
- Declined to comment for OregonLive story 1
- Courtland Geyer, Marion County Circuit Court Judge
- Handling the primary case it looks like.
- Ordered both sides into mediation.
- Carolyn Lawson, former chief information officer for the Oregon Health Authority
- Kyle Busse, one of Lawson's attorneys in her lawsuit against the state
- David Angeli, Portland lawyer who also represented Lawson
- Rocky King, Head of Cover Oregon
- Clyde Hamstreet, interim Executive Director of Cover Oregon
- Alex Pettit, the state’s top IT manager
- In his resignation letter to Governer Kate Brown he says, "We settled the Oracle dispute, offering the state the use of nearly all Oracle software products."
- So is that why they settled for ongoing services from Oracle? They were holding the other products hostage until they settled?
- Willamette Week article about his resignation - https://www.wweek.com/news/2018/04/09/oregons-chief-information-officer-alex-pettit-resigns/
- In his resignation letter to Governer Kate Brown he says, "We settled the Oracle dispute, offering the state the use of nearly all Oracle software products."
- Bruce Goldberg, long-time government health chief
- Resigned as part of the fallout.
- Bob Stoll, court appointed mediator
- "Strangely, Geyer appointed Bob Stoll as mediator. A prominent retired lawyer who founded the Stoll Berne firm in Portland, Stoll is a seasoned attorney. But he has little experience as a mediator. What's more, he's perhaps the most prominent Democratic activist and fundraiser in the state. He's donated thousands of dollars to Brown over the years." ~ OregonLive story 1
- Dave Markowitz, Lawyer
- "the prominent Portland lawyer whose firm is leading the Oracle litigation and has collected most of the $6 million-plus the state has spent to date on the case." ~ OregonLive story 1
- Ken Glueck, Oracle senior vice president and its pointman in the Oregon litigation
- Safra Katz, Oracle's president and chief financial officer
- Named defendants in the litigation.
- Dorian Daley, Oracle's general counsel
- Deborah Hellinger, Oracle spokesperson
- Claims the State cherry picked bad quotes out of the 2 million documents. ~ PortlandTribune1
- Tung Yin, a Lewis & Clark Law School professor
- "“The worst possible reading of that email would be, I think, that there were people who knew (the website) would never work, but they were still selling it to Oregon and taking the money,” Yin says." ~ PortlandTribune1
- Todd Williams, a management consultant and author of a book on problem projects
- "“I do believe Oregon is much more at fault,” Williams says. “Oregon screwed its population on this project.”" ~ PortlandTribune1
- Tom Walsh, longtime technology specialist and veteran of the Cover Oregon project
- Quoted extensively in this Portland Tribune article.
- "One of the roving breed of professional consultants who bounce from state to state for months or years at a time, Walsh is typically the top-dog “systems analyst” who either leads or troubleshoots large IT project design or is paid to watchdog those who do, says Shari Benkiel, a longtime IT consultant who has worked with Walsh on seven large projects in five states." ~ Portland Tribune article
- "Walsh applied and went to work on it in April 2012 as a consultant." ~ Portland Tribune article
- Shari Benkiel, worked with Walsh
- Louisa Moore, worked with Walsh
- "Cover Oregon" was a DBA for OREGON HEALTH INSURANCE EXCHANGE CORPORATION, a Public Corporation. From the legal complaint filed by Oracle.
- Remember when Last Tonight did this - https://www.youtube.com/watch?v=Dh9munYYoqQ