Python nltk.clean_htmlは実装されていません

Question

使ってみました

myNews=urlopen(url).read() myNews=nltk.clean_html(myNews)

次のエラーが発生します。

Clean_htmlでファイル「/usr/local/lib/python2.7/dist-packages/nltk-3.0.0-py2.7.Egg/nltk/util.py」の346行目をNotImplementedErrorにレイズ（「HTMLマークアップを削除するには、 BeautifulSoupのget_text（）関数を使用してください "）NotImplementedError：HTMLマークアップを削除するには、BeautifulSoupのget_text（）関数を使用してください

ファイルutil.pyを見ると、実装されていないことがわかります。

def clean_html(html): raise NotImplementedError ("To remove HTML markup, use BeautifulSoup's get_text() function")

実装しないでください。

alvas · Accepted Answer

clean_html()とclean_url()は、NLTKのかわいい関数です。BeautifulSoupがより優れた仕事とマークアップ言語の解析を行うため、ドロップされました https://github.com/nltk/を参照） nltk/commit/39a303e5ddc4cdb1a0b00a3be426239b1c24c8bb

BeautifulSoupのドキュメントは次のとおりです。 http://www.crummy.com/software/BeautifulSoup/bs4/doc/

Michael · Answer

他の回答ノートとして、ntlk この機能は削除されましたで、「HTMLマークアップを削除するには、BeautifulSoupのget_text（）関数を使用する」ことをお勧めします。 Beautiful Soupは、特定の要素からテキストを抽出する場合に適していますが、ページ全体のテキストが必要な場合は、nltk関数を使用してください。 2つのアプローチの比較を次に示します。

import mechanize import nltk from bs4 import BeautifulSoup from html2text import html2text import re def clean_html(html): """ Copied from NLTK package. Remove HTML markup from the given string. :param html: the HTML string to be cleaned :type html: str :rtype: str """ # First we remove inline JavaScript/CSS: cleaned = re.sub(r"(?is)<(script|style).*?>.*?(</\1>)", "", html.strip()) # Then we remove html comments. This has to be done before removing regular # tags since comments can contain '>' characters. cleaned = re.sub(r"(?s)<!--(.*?)-->[
]?", "", cleaned) # Next we can remove the remaining tags: cleaned = re.sub(r"(?s)<.*?>", " ", cleaned) # Finally, we deal with whitespace cleaned = re.sub(r"&nbsp;", " ", cleaned) cleaned = re.sub(r" ", " ", cleaned) cleaned = re.sub(r" ", " ", cleaned) return cleaned.strip() url = "http://www.nytimes.com/2015/08/31/business/challenged-on-left-and-right-the-fed-faces-a-decision-on-rates.html" br = mechanize.Browser() br.set_handle_robots(False) br.addheaders = [('User-agent', 'Firefox')] html = br.open(url).read().decode('utf-8') cleanhtml = clean_html(html) text = html2text(cleanhtml) soup = BeautifulSoup(html) text2 = soup.get_text()

Nltk関数を使用すると、Nice cleanの結果が得られます（ここを参照、投稿が最大30,000文字を超えたため、投稿するにはPastebinに入れる必要がありました）。そして、美しいスープ：

u'
 




Challenged on Left and Right, the Fed Faces a Decision on Rates - The New York Times
window.NREUM||(NREUM={}),__nr_require=function(n,e,t){function r(t){if(!e[t]){var o=e[t]={exports:{}};n[t][0].call(o.exports,function(e){var o=n[t][1][e];return r(o?o:e)},o,o.exports)}return e[t].exports}if("function"==typeof __nr_require)return __nr_require;for(var o=0;o<t.length;o++)r(t[o]);return r}({QJf3ax:[function(n,e){function t(n){function e(e,t,a){n&&n(e,t,a),a||(a={});for(var u=c(e),f=u.length,s=i(a,o,r),p=0;f>p;p++)u[p].apply(s,t);return s}function a(n,e){f[n]=c(n).concat(e)}function c(n){return f[n]||[]}function u(){return t(e)}var f={};return{on:a,emit:e,create:u,listeners:c,_events:f}}function r(){return{}}var o="nr@context",i=n("gos");e.exports=t()},{gos:"7eSDFh"}],ee:[function(n,e){e.exports=n("QJf3ax")},{}],3:[function(n,e){function t(n){return function(){r(n,[(new Date).getTime()].concat(i(arguments)))}}var r=n("handle"),o=n(1),i=n(2);"undefined"==typeof window.newrelic&&(newrelic=window.NREUM);var a=["setPageViewName","addPageAction","setCustomAttribute","finished","addToTrace","inlineHit","noticeError"];o(a,function(n,e){window.NREUM[e]=t("api-"+e)}),e.exports=window.NREUM},{1:12,2:13,handle:"D5DuLP"}],gos:[function(n,e){e.exports=n("7eSDFh")},{}],"7eSDFh":[function(n,e){function t(n,e,t){if(r.call(n,e))return n[e];var o=t();if(Object.defineProperty&&Object.keys)try{return Object.defineProperty(n,e,{value:o,writable:!0,enumerable:!1}),o}catch(i){}return n[e]=o,o}var r=Object.prototype.hasOwnProperty;e.exports=t},{}],D5DuLP:[function(n,e){function t(n,e,t){return r.listeners(n).length?r.emit(n,e,t):(o[n]||(o[n]=[]),void o[n].Push(e))}var r=n("ee").create(),o={};e.exports=t,t.ee=r,r.q=o},{ee:"QJf3ax"}],handle:[function(n,e){e.exports=n("D5DuLP")},{}],XL7HBI:[function(n,e){function t(n){var e=typeof n;return!n||"object"!==e&&"function"!==e?-1:n===window?0:i(n,o,function(){return r++})}var r=1,o="nr@id",i=n("gos");e.exports=t},{gos:"7eSDFh"}],id:[function(n,e){e.exports=n("XL7HBI")},{}],loader:[function(n,e){e.exports=n("G9z0Bl")},{}],G9z0Bl:[function(n,e){function t(){var n=h.info=NREUM.info;if(n&&n.licenseKey&&n.applicationID&&f&&f.body){c(l,function(e,t){e in n||(n[e]=t)}),h.proto="https"===d.split(":")[0]||n.sslForHttp?"https://":"http://",a("mark",["onload",i()]);var e=f.createElement("script");e.src=h.proto+n.agent,f.body.appendChild(e)}}function r(){"complete"===f.readyState&&o()}function o(){a("mark",["domContent",i()])}function i(){return(new Date).getTime()}var a=n("handle"),c=n(1),u=(n(2),window),f=u.document,s="addEventListener",p="attachEvent",d=(""+location).split("?")[0],l={beacon:"bam.nr-data.net",errorBeacon:"bam.nr-data.net",agent:"js-agent.newrelic.com/nr-593.min.js"},h=e.exports={offset:i(),Origin:d,features:{}};f[s]?(f[s]("DOMContentLoaded",o,!1),u[s]("load",t,!1)):(f[p]("onreadystatechange",r),u[p]("onload",t)),a("mark",["firstbyte",i()])},{1:12,2:3,handle:"D5DuLP"}],12:[function(n,e){function t(n,e){var t=[],o="",i=0;for(o in n)r.call(n,o)&&(t[i]=e(o,n[o]),i+=1);return t}var r=Object.prototype.hasOwnProperty;e.exports=t},{}],13:[function(n,e){function t(n,e,t){e||(e=0),"undefined"==typeof t&&(t=n?n.length:0);for(var r=-1,o=t-e||0,i=Array(0>o?0:o);++r<o;)i[r]=n[e+r];return i}e.exports=t},{}]},{},["G9z0Bl"]);

































































































{"pageconfig":{"ledeMediaSize":"large","keywords":["article-medium","has-embedded-interactive"]}}

 [] 

var googletag=googletag||{};googletag.cmd=googletag.cmd||[],function(){var t=document.createElement("script");t.async=!0,t.type="text/javascript";t.src="http://www.googletagservices.com/tag/js/gpt.js";var o=document.getElementsByTagName("script")[0];o.parentNode.insertBefore(t,o)}();


[
 {
 "testId": "0012",
 "testName": "tallWatchingModule",
 "throttle": 1.0,
 "allocation": 0.9,
 "variants": 1,
 "applications": ["homepage"]
 },
 {
 "testId": "0033",
 "testName": "recommendedLabelTest",
 "throttle": 1,
 "allocation": 0.833,
 "variants": 5,
 "applications": ["article"]
 },
 {
 "testId": "0036",
 "testName": "velcroSocialFollow",
 "throttle": 0.1,
 "allocation": 0.5,
 "variants": 1,
 "applications": ["article", "homepage"]
 },
 {
 "testId": "0050",
 "testName": "styledMostEmailed",
 "throttle": 1,
 "allocation": 0.667,
 "variants": 2,
 "applications": ["article"]
 },
 {
 "testId": "0051",
 "testName": "shuffleRecommendations",
 "throttle": 1.0,
 "allocation": 0.667,
 "variants": 1,
 "applications": ["article"]
 },
 {
 "testId": "0052",
 "testName": "paidPostDriver",
 "throttle": 1.0,
 "allocation": 0.875,
 "variants": 7,
 "applications": ["article"]
 },
 {
 "testId": "0061",
 "testName": "paidPostFivePack",
 "throttle": 0,
 "allocation": 0,
 "variants": 1,
 "applications": ["homepage"]
 }
]



{ "meta": {},
 "data": {
 "id": "0",
 "name": "",
 "subscription": ["","_RPV"],
 "demographics": {}
 }
}


var require = {
 baseUrl: \'http://a1.nyt.com/assets/\',
 waitSeconds: 20,
 paths: {
 \'foundation\': \'article/20150828-192044/js/foundation\',
 \'shared\': \'article/20150828-192044/js/shared\',
 \'article\': \'article/20150828-192044/js/article\',
 \'application\': \'article/20150828-192044/js/article/article\',
 \'videoFactory\': \'http://static01.nyt.com/js2/build/video/2.0/videofactoryrequire\',
 \'videoPlaylist\': \'http://static01.nyt.com/js2/build/video/players/extended/2.0/appRequire\',
 \'auth/mtr\': \'http://static01.nyt.com/js/mtr\',
 \'auth/growl\': \'http://static01.nyt.com/js/auth/growl/default\',
 \'vhs\': \'http://static01.nyt.com/video/vhs/build/vhs-2.x.min\'
 },
 map: {
 \'*\': {
 \'article/main\': \'article/article/main\'
 }
 }
};






window.magnum.processFlags(["limitFabrikSave","moreFollowSuggestions","dfpAds","dfpWhitelist","criticsPickAdditionalInfo","restaurantAttributes","theaterAttributes","movieAttributes","followFeature","restaurantReviewAdditionalDetails","theaterReviewAdditionalDetails","restaurantReviewHideInfoBox","theaterReviewHideInfoBox","restaurantReviewShowRestaurantName","restaurantReviewShowGoogleMap","restaurantReviewShowNotes","restaurantReviewShowLastUpdated","styledMostEmailed","videoVHSCover","restaurantReviewShowMenuLink","allTheEmphases","androidDeepLinks","autoPlayVideos","restaurantOpenStatus","standaloneSlideshowPromo","showNewTMagLogo"]);


require([\'foundation/main\'], function () {
 require([\'auth/mtr\', \'auth/growl\']);
});




 .lt-ie10 .messenger.suggestions {
 display: block !important;
 height: 50px;
 }

 .lt-ie10 .messenger.suggestions .message-bed {
 background-color: #f8e9d2;
 border-bottom: 1px solid #ccc;
 }

 .lt-ie10 .messenger.suggestions .message-container {
 padding: 11px 18px 11px 30px;
 }

 .lt-ie10 .messenger.suggestions .action-link {
 font-family: "nyt-franklin", arial, helvetica, sans-serif;
 font-size: 10px;
 font-weight: bold;
 color: #a81817;
 text-transform: uppercase;
 }

 .lt-ie10 .messenger.suggestions .alert-icon {
 background: url(\'http://i1.nyt.com/images/icons/icon-alert-12x12-a81817.png\') no-repeat;
 width: 12px;
 height: 12px;
 display: inline-block;
 margin-top: -2px;
 float: none;
 }

 .lt-ie10 .masthead,
 .lt-ie10 .navigation,
 .lt-ie10 .comments-panel {
 margin-top: 50px !important;
 }

 .lt-ie10 .ribbon {
 margin-top: 97px !important;
 }






NYTimes.com no longer supports Internet Explorer 9 or earlier. Please upgrade your browser.
LEARN MORE \xbb









Sections

Home

Search
Skip to content
Skip to navigation
View mobile version




The New York Times


window.magnum.writeLogo(\'small\', \'http://a1.nyt.com/assets/article/20150828-192044/images/foundation/logos/\', \'business\', \'masthead-theme-standard\', \'standard\', \'branding-heading-link\');


Economy|Challenged on Left and Right, the Fed Faces a Decision on Rates



Advertisement







Search


Log In
0
Settings




Close search

search sponsored by






Search NYTimes.com



Clear this text input



Go






http://nyti.ms/1VpLa1D




Loading...




See next articles





See previous articles








 







Advertisement






Economy 
Challenged on Left and Right, the Fed Faces a Decision on Rates

By BINYAMIN APPELBAUMAUG. 30, 2015


Inside



Supported by








Photo






Janet L. Yellen, the Federal Reserve chairwoman.

Credit
 Stephen Crowley/The New York Times 




Advertisement

Continue reading the main story





Continue reading the main story
Share This Page

Continue reading the main story


Continue reading the main story



JACKSON HOLE, Wyo. \u2014 Conservative activists who want the Federal Reserve to raise interest rates distributed chocolate coins in golden wrappers at the local airport last week as Fed officials arrived for their annual policy retreat.Liberal activists in green \u201cWhose Recovery?\u201d T-shirts formed a receiving line at the resort hotel in the heart of Grand Teton National Park where the meeting was held, to personalize their argument that the Fed should wait.Sometime soon \u2014 possibly as early as mid-September and probably no later than the end of the year \u2014 the Fed plans to raise its benchmark interest rate one-quarter of one percentage point, a mathematically minor move that has become a very big deal.Investors, who always pay attention to the Fed, are paying particular attention now. The central bank has held short-term rates near zero since December 2008; the impending end of that era is one cause of recent financial market turmoil. 

Continue reading the main story


 Related Coverage
 










Optimistic About Inflation, Stanley Fischer Suggests That Fed Will Stick to Plan on RatesAUG. 29, 2015







But the Fed\u2019s plans have also become the latest point of contention in a broader debate about the government\u2019s management of the American economy, pitting liberals who see a need for more aggressive measures to bolster growth against conservatives concerned that Washington and the Fed are already doing much too much. 
Continue reading the main story



 When Will the Fed Raise Rates? 

 More than seven years ago the Federal Reserve put its benchmark interest rate close to zero, as a way to bolster the economy. But that policy is about to change. 











\u201cThere shouldn\u2019t be this intense interest in a quarter-point increase, and there shouldn\u2019t be this intense interest in whether it comes in September or December,\u201d said Alan S. Blinder, a Princeton economist and the Fed\u2019s vice chairman in the mid-1990s. \u201cBut the Fed remains the center of the financial universe. People stare at it like they stare at the North Star.\u201dAnd so, as Fed officials conferred with other central bankers and academics, the liberal activists held two days of \u201cFed Up\u201d teach-ins in a room directly below the main conference, while the conservatives convened a \u201cJackson Hole Summit\u201d at a nearby dude ranch.In the decades before the financial crisis, policy makers generally agreed that central banks should focus on moderating inflation. Now, both that goal and the best way to achieve it are subjects of debate. Liberals argue that the Fed should aim more broadly to lower unemployment and encourage rising living standards. Conservatives want to strengthen the focus on inflation by requiring officials to follow rules in making policy.
Advertisement

Continue reading the main story
With the critics lining up outside, central bankers found no escape inside the main conference, where a series of academics warned policy makers that their view of inflation was oversimplified, and that their policies were less effective as a consequence.\u201cThe conference was more about what we don\u2019t know, about a candid willingness to analyze what we don\u2019t know,\u201d said Lucrezia Reichlin, a professor at London Business School and former director general of research at the European Central Bank. \u201cIt did not really inspire confidence\u201d in monetary policy.The formal program, on \u201cInflation Dynamics and Monetary Policy,\u201d was devoted to the vexing reality that inflation in recent years has not behaved as economists predicted. The basic paradigm, known as the Phillips Curve, is that inflation falls as unemployment rises, and rises as unemployment falls. But inflation did not fall as much as expected during the Great Recession, and it has remained surprisingly weak during the recovery.
Advertisement

Continue reading the main story
Over the course of two days, the invited academics argued that the real story was more complicated. One study, for example, presented evidence that prices fall more slowly during recessions because cash-short firms actually tend to increase prices in the face of declining demand for their products.\u201cOnce you integrate all these dynamics, it may turn out that life is not that simple,\u201d said Eric M. Leeper, an economist at Indiana University and co-author of a paper arguing that central banks need better economic models.Central bankers, however, have shown little interest in paradigm shifts. Several said that the basic understanding of inflation, while obviously imperfect, remains more functional than any alternatives.\u201cI don\u2019t think the folks at the Fed are of a mind to redesign monetary policy just because of what happened during the crisis,\u201d said Jon Faust, a professor of economics at Johns Hopkins University and a former adviser to the Fed\u2019s chairwoman, Janet L. Yellen, and her predecessor, Ben S. Bernanke.Indeed, V\xedtor Const\xe2ncio, vice president of the European Central Bank, said the euro area was currently experiencing \u201ca renaissance of the Phillips Curve.\u201dStanley Fischer, vice chairman of the Federal Reserve, painted a somewhat more complicated picture of inflation, arguing that the role of labor market slack is easily overstated, and that exchange rates play an important role.
Continue reading the main story
Video

The Fed\u2019s Button on the Economy

When it comes to raising or lowering interest rates, what the Fed is really trying to do is balance growth and inflation. But they have a limited set of tools to accomplish their goal.

 By Andrew Ross Sorkin, Aaron Byrd and Erica Berenstein on Publish Date July 29, 2015.
 

 Photo by Aaron Byrd/The New York Times.
 
Watch in Times Video \xbb



But his bottom line, too, was that the Fed understands inflation well enough to predict its movements. While domestic inflation has been surprisingly sluggish for years now, Mr. Fischer said on Friday that his confidence in an eventual rebound remained \u201cpretty high.\u201dThe organizers of the fringe conferences acknowledged the odds against their more radical proposals.\u201cFed Up\u201d is mostly funded by the foundation of a Facebook co-founder, Dustin Moskovitz, which said: \u201cOur best guess is that the campaign is unlikely to have an impact on the Fed\u2019s monetary policy, but that if it does, the benefits would be very large.\u201dJim DeMint, president of the Heritage Foundation, spoke at the conservative conference of \u201ca long and difficult battle that we can and must win.\u201dThe Center for Public Democracy, which organized the \u201cFed Up\u201d campaign, wants the Fed to keep rates near zero even as overall unemployment falls, to spur wage gains and help members of minorities, in particular, find jobs. It brought about 50 people to Jackson Hole as part of an effort to engage community groups that generally focus on civil rights or local issues like minimum wage laws.Dawn O\u2019Neal, 48, makes $8.50 an hour as a day care worker in suburban Atlanta; her husband has not found regular construction work in a year. When Ms. O\u2019Neal needs a refill on her asthma medication, she cuts back on food, buying hot dogs instead of beef and canned vegetables instead of fresh vegetables.\u201cI don\u2019t feel like anyone at the Fed has ever had to make a decision about whether to eat or get medication, and so when I hear that they\u2019re going to raise interest rates in September, it angers me and it scares me,\u201d Ms. O\u2019Neal said.
Advertisement

Continue reading the main story


Advertisement

Continue reading the main story
The protesters struck a chord with some officials at the main meeting. Jason Furman, President Obama\u2019s chief economic adviser, went downstairs and delivered an impromptu speech. \u201cWe don\u2019t comment on monetary policy, but what I can say is that monetary policy matters,\u201d he told the activists. The prosperity of the late 1990s, he added, resulted in part from \u201ca set of decisions made by the Federal Reserve that allowed that to happen.\u201dOther officials, however, said the Push for low rates was misguided.\u201cThe biggest risk for those that are less fortunate is that we would go back into recession,\u201d said James Bullard, president of the Federal Reserve Bank of St. Louis, who said he leaned toward raising rates in September. \u201cI\u2019m hoping my policy would lengthen out the expansion longer.\u201dThe conservative conference was aligned with efforts by congressional Republicans to impose new restrictions on the Fed\u2019s conduct of monetary policy. A leading proposal would require the Fed to choose a formula for setting rates and stick with it.This view has few fans among the central bankers, who see their own judgment as an essential part of policy making.Mr. Blinder said part of the disconnect between the officials and the activists may reflect that broader concerns motivate liberals and conservatives. Conservatives see the Fed as enabling the growth of the federal debt, while liberals see the Fed as contributing to the rise of inequality.Mr. Blinder said the central bank had little power to reverse either trend. \u201cThey overstate the importance and power of the Federal Reserve,\u201d he said. All it can do, he added, is \u201caddress these problems around the edges.\u201d


A version of this article appears in print on August 31, 2015, on page A1 of the New York edition with the headline: Left and Right Work to Shift Fed\u2019s Direction. Order Reprints| Today\'s Paper|Subscribe











Loading...









Go to Home Page \xbb

Site Index

The New York Times


window.magnum.writeLogo(\'small\', \'http://a1.nyt.com/assets/article/20150828-192044/images/foundation/logos/\', \'\', \'\', \'standard\', \'site-index-branding-link\');




News


World


U.S.


Politics


N.Y.


Business


Tech


Science


Health


Sports


Education


Obituaries


Today\'s Paper


Corrections




Opinion


Today\'s Opinion


Op-Ed Columnists


Editorials


Contributing Writers


Op-Ed Contributors


Opinionator


Letters


Sunday Review


Taking Note


Room for Debate


Public Editor


Video: Opinion




Arts


Today\'s Arts


Art & Design


ArtsBeat


Books


Dance


Movies


Music


N.Y.C. Events Guide


Television


Theater


Video Games


Video: Arts




Living


Automobiles


Crossword


Food


Education


Fashion & Style


Health


Jobs


Magazine


N.Y.C. Events Guide


Real Estate


T Magazine


Travel


Weddings & Celebrations




Listings & More


Classifieds


Tools & Services


Times Topics


Public Editor


N.Y.C. Events Guide


TV Listings


Blogs


Cartoons


Multimedia


Photography


Video


NYT Store


Times Journeys


Subscribe


Manage My Account




Subscribe

Subscribe


Times Premier



Home Delivery



Digital Subscriptions



NYT Opinion



Crossword




Email Newsletters


Alerts


Gift Subscriptions


Corporate Subscriptions


Education Rate




Mobile Applications


Replica Edition


International New York Times











 \xa9 2015 The New York Times Company


Home
Search
Contact Us
Work With Us
Advertise
Your Ad Choices
Privacy
Terms of Service
Terms of Sale




Site Map
Help
Site Feedback
Subscriptions






require([\'foundation/main\'], function () {
 require([\'article/main\']);
 require([\'jquery/nyt\', \'foundation/views/page-manager\'], function ($, pageManager) {
 if (window.location.search.indexOf(\'disable_tagx\') > 0) {
 return;
 }
 $(document).ready(function () {
 require([\'http://static01.nyt.com/bi/js/tagx/tagx.js\'], function () {
 pageManager.trackingFireEventQueue();
 });
 });
 });
});










window.NREUM||(NREUM={});NREUM.info={"beacon":"bam.nr-data.net","licenseKey":"b5bcf2eba4","applicationID":"4491457","transactionName":"YwFXZhRYVhAEVUZcX1pLYEAPFlkTFRhCXUA=","queueTime":0,"applicationTime":305,"ttGuid":"","agentToken":"","userAttributes":"","errorBeacon":"bam.nr-data.net","agent":"js-agent.newrelic.com\/nr-593.min.js"}

'

スクロールするとわかるように、Beautiful Soupバージョンには目に見えないテキストがたくさん含まれています。あまりきれいではありません。

Statham · Answer

あなたのコードが

raw = nltk.clean_html(html) tokens = nltk.Word_tokenize(raw)

使用できます

raw = BeautifulSoup(html).get_text() tokens = nltk.Word_tokenize(raw)

代わりに、理由について他の回答を参照してください。